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Executive Summary 


Through Delaware's Positive Behavior Supports initiative (DE-PBS), over one hundred public schools in 
Delaware have adopted the Schoolwide Positive Behavioral Supports approach to schoolwide discipline 
(Sugai & Horner, 2009). This approach focuses on developing a school-wide system of strategies to 
reduce behavior problems and foster a positive school climate. DE-PBS is a collaborative effort between 
the Delaware Department of Education (DDOE), the Center for Disabilities Studies at the University of 
Delaware, and Delaware public schools. The project is led by a PBS Core Team consisting of project staff 
members from the Center for Disabilities Studies (a project co-director and a research associate), the 
Delaware Department of Education (a DDOE staff member who serves as co-director). University of 
Delaware faculty partners (two professors in the School of Education who serve as consultants), and a 
consulting school psychologist who works in the public schools. In previous years, the Core Team has 
focused on increasing the number of schools implementing PBS and developing staff development 
materials. Although PBS entails 3 tiers of supports and interventions (Tier 1 for all students. Tier 2 for at- 
risk students, and Tier 3 for students with the most serious behavior problems), the primary focus of DE- 
PBS has been at Tier 1 - the universal, schoolwide level. 


The number of DE-PBS schools has more than doubled in the past five years. Although many more 
schools are now designated as PBS schools, the rapid increase was associated with variability in the 
quality of program implementation across schools. To address the implementation issues, and increase 
the likelihood that the program will have the desired effects, the project has focused on three critical 
tasks in recent years. These tasks are: (1) developing key features of DE-PBS, including the valued 
student outcomes and the processes for obtaining them, (2) creating reliable and valid evaluation 
measures of those outcomes and processes, and (3) aligning key features and assessment measures 
with staff development and training. DE-PBS is currently in a state of transition - evolving from a 
primarily behaviorally-oriented approach to managing student behavior to a more comprehensive 
approach to schoolwide discipline designed not only to help manage student behavior but also to 
develop self-discipline and foster positive school climate. 

Evaluation Questions 

Two questions guided the evaluation. The first is formative, intended to help the DE-PBS Core Team in 
its current transition. The second looks again at the outcome of disciplinary infractions, using a different 
analysis than that used by Bear and colleagues (2009) to see if the results are any different. The specific 
questions are: 

(1) What are contributors and barriers to schools' implementation of DE-PBS? 

(2) What is the relationship between DE-PBS and changes in incidence of disciplinary infractions? 

Contributors and Barriers to DE-PBS Implementation 

To learn about contributors and barriers to PBS implementation, a multiple case study design with six 
schools was used. To be considered for inclusion in the study, schools had to: (1) be an elementary or 
middle school implementing DE-PBS for at least three years, (2) have the approval of district leaders to 
participate, and (3) be at either the high or low end of the continuum of school disciplinary infractions. 
We purposively selected schools at the ends of the continuum to ensure a diversity of experiences in the 
implementation of PBS. The final sample includes three middle schools and three elementary schools, at 
least one school from each county, and two schools with a high rate of disciplinary infractions and four 



with low rates. The participating schools are not identified because the focus is on the cross-cutting 
findings on barriers and contributors, rather than on successes or challenges at specific schools. 

At each school, administrators serving on the PBS team participated in individual interviews. All 
members of a school's PBS team were invited to participate in a focus group interview. Focus groups 
were also held with teachers and, in the schools with low rates of disciplinary infractions, with students. 
(Given time constraints, focus groups could not be held with students at all schools.) In all but one 
school, participants in the teacher and student focus groups were randomly selected from lists provided 
by the schools. (One principal chose to select teachers and students by following our random selection 
procedures.) With the exception of the students, all interviewees were asked to complete an online 
version of the Delaware Assessment of Strengths and Needs for Positive Behavior Supports (Bear, 
Burwell, Baker, Blank, & Boyer, 2009), which includes items aligned with features of DE-PBS and drawn 
from the literature. Respondents from schools with higher incidence of disciplinary infractions were 
more likely to give their school low ratings than respondents from schools with lower incidence of 
infractions. These results indicate the diversity of the selected schools on dimensions relevant to PBS 
implementation, and thus confirm the schools' likely utility in providing different perspectives on DE- 
PBS. 

The data from the individual and focus group interviews were transcribed from audio recordings and 
then analyzed using a combination of deductive categories drawn from the key features of DE-PBS and 
inductive coding to identify emergent themes. In keeping with the focus on cross-cutting contributors 
and barriers rather than on the experiences of individual schools, the analysis was conducted by type of 
respondent first (that is, all PBS team focus groups, then all administrators, and so on) and then the 
results were aggregated across the different respondents to identify major categories and specific 
contributors and barriers. This approach limits the opportunity to include the role of school context in 
the analysis. The goal of this part of the evaluation was to help the team learn how to improve school's 
implementation of PBS, whatever its context. 

Findings 

Four dominant themes related to contributors and barriers to the implementation of DE-PBS were 
identified: 

(1) consistency and adaptability, 

(2) rewards, 

(3) data-based decision making, and 

(4) professional development and support. 

These themes encompass both contributors and barriers to PBS implementation. For example, the 
theme of consistency and adaptability includes the positive aspect of PBS evolving over time in response 
to changing needs, and the concern about PBS becoming stagnant and unresponsive. 
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Consistency and Adaptability 


Consistency and adaptability, the most prominent factor identified in the data, has three dimensions: 
the consistency of commitment to PBS oyer time; the consistency with which school-specific PBS policies 
and practices were applied within each school; and the adaptability of PBS as part of a dynamic and 
eyolying school culture. Table 1 proyides illustratiye quotes for the contributors and barriers associated 
with this theme. In this and the other tables in this section, the initials in brackets after each quote refer 
to the source of the quote, with C = coach, A = administrator, T = teacher (not a member of the PBS 
team), S = student, and TM = PBS team member. 


Table 1. Contributors and barriers in the consistency and adaptability 


Contributors 

Illustrative Quotes 

Consistent commitment to PBS by 
teachers and administrators 

"They have to have buy-in - that's important so you have to get people to 
understand why it's important and why you're doing it." [A] 

Consistency of policies and practices 
as applied in the school 

"Those rules are throughout the whole school, not only in the classroom, 
but it's being safe in the hallway, in the playground, on the bus, and in the 
cafeteria. ..The rules are the same no matter where you are and no matter 
what grade you are in." [T] 

Willingness to adapt and experiment 
within the framework of PBS 

"So 1 think one thing we're always doing is trying to say like 'where are 
we?' and 'what can we do to improve it?"' [A] 

Barriers 

Illustrative Quotes 

Lack of buy-in to PBS 

"There are still considerable people, teachers and staff, who are resistant 
and that clearly affects how well they are able to implement. There are 
some naysayers that clearly don't want to be on board." [C] 

Limited understanding of PBS 
principles 

"...something we're constantly working on is people understanding it [the 
currency used as rewards] has to be one at a time. .That's an education 
that goes on every month it seems like. Being sure that people use them 
the way they're supposed to be used." [A] 


Rewards 

When asked to describe the major components of PBS in their school, almost all respondents mentioned 
the rewards system before any other element of the program. The structure of rewards systems was 
similar across the schools. Each school had a currency system in place wherein students were issued 
yarying amounts of the school currency for different types of positiye behayior. Although the currency 
was a common element, there was yariation in how it was used. In different schools, students could use 
the currency to buy pre-priced items in a PBS store, bid on items at auctions, participate in a weekly 
lottery, or purchase extra priyileges. In some cases, schools used a combination of group actiyities and 
eyents with indiyidual prizes. In the six schools that we studied, rewards seem to dominate the 
definition of PBS. 

The contributors and barriers associated with the theme of rewards are listed below, with illustratiye 
quotes for each. 


y 




Table 2. Contributors and barriers in the rewards theme 


Contributors 

Illustrative Quotes 

Student input 

"With the school store it has changed because of the survey that was 
taken by fifth grade students and they came up with different things they 
would like to see in the school store." [T] 

Appropriate nature and timing of 
rewards 

"Another thing that is always told to us is to keep the rewards simple 
because you don't want the students thinking that if I'm good 1 can get a 
reward. You want them to think 1 should do the right thing. So we always 
try to keep the rewards really simple." [TM] 

External support 

"Another example of a donation or a pitch would be to go to the local 
WaWa and students that have the best behaved classroom. ..for that 
month, they would receive a free milkshake from WaWa." [A] 

Barriers 

Illustrative Quotes 

Lack of clarity/agreement about 
when to use rewards 

"...1 feel as though not every teacher gives the students the tickets for the 
same behaviors. It's not consistent among teachers." [TM] 

Student manipulation of the system 

"1 just feel that as the kids are going from grade to grade they're learning 
how to manipulate the system, so by fifth grade 1 mean I've had a kid come 
up to me and say put his feet on my desk and say let's make a deal." [T] 

Rewards that are inappropriate for 
the age group 

"The rewards are more for like middle school interests. Like in middle 
school they're not going to want to be winning candy, they're going to 
want to win money and stuff that they can actually use." [S] 


Data-Based Decision Making 

Data-based decision making is a key feature of PBS and was addressed specifically in our interview 
questions. In response, teachers and administrators described positive aspects of using data, but also 
identified barriers to the use of data. Both positive aspects and barriers, with illustrative quotes for 
each, are listed in the following table. 


Table 3. Contributors and barriers in the data-based decision-making theme 


Contributors 

Illustrative Quotes 

Responding to emerging issues 

"Say we noticed in February that the playground behavior was poor, it was 
going down and we were having incidents, then we would go back and 
discuss with our co-workers what we need to do, possibly go over the 
rules, do some more skits, remind the kids again about the PBS rules, show 
them again what they look like, and model again good behavior and what 
we're expecting out of them." [TM] 

Monitoring individual behavior 

"We look at our monthly offenses (level 2, 3, 4). We look at where they are 
occurring, are there repeat offenders, and what have we done for those 
repeat offenders. Are they seeing the guidance counselor, so they have an 
FBA (Functional Behavior Assessment), do they have an lEP that has 
accommodations. Are we meeting those?" [A] 

Increased staff buy-in 

"But some of our teachers weren't buying into it as much as they could 
have for a lot of reasons. One is because they didn't see the benefits of it. 
Well, now they see the benefits of it because we are able to track it and 
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get a little bit better, and we are able to use it a little bit better, so now we 
have more teachers buying into it." [A] 

Barriers 

Illustrative Quotes 

Data as window-dressing 

"...we have the academic data posted all around in our classroom, down 
stairs, when you walk in the buildings, you hear about it during staff 
development meetings, but we never hear about the school climate data. 
And the referrals, that data is not posted." [T] 

Unsystematic collection of data 

"1 would ask them - where's your data? First, it was not even there to 
begin with. Then they started bringing data, but 1 think that they thought 
all they had to do was look at it instead of analyze it. Now 1 think they are 
getting it; it's emerging. 1 think they're improving, but that has always 
been a major roadblock in their ability to reflect on what they're doing and 
make decisions based on the data about what needs to be done 
differently." [C] 


Professional Development and Support 

Schoolwide and targeted subgroup professional development and support was a theme discussed 
among all adult participant groups. Need for professional development varied widely across schools and 
refresher training was identified as an important part of the process. Support from the district coach 
also varied with some schools reporting that they needed little assistance and others needing more. The 
quotes below illustrate the contributors and barriers related to professional development and support. 


Table 4. Contributors and barriers in the professional development and support theme 


Contributors 

Illustrative Quotes 

Awareness of District Coach as a 
resource 

"At this point we felt we haven't needed our district coach, she's there if 
we need her, but things have been running smoothly. " [TM] 

Annual refresher training 

"We actually revamped our program. ..and so we went to a retreat where 
we spent a good part of the day re-training all the new staff so everybody 
had a refresher." [A] 

Barriers 

Illustrative Quotes 

Insufficient training for new teachers 

"I'm a first year teacher here and I've never gotten any training. 1 spent the 
first three weeks, they gave me tickets [reward currency], and 1 had no 
idea what they were." [T] 

Lack of refresher training 

"1 think it is necessary every year to cover the whole thing - the basics, the 
philosophy, the way it works. ..There is some time that should be invested 
in making it work every year and we just haven't been able to put the time 
into because of all the other constraints in the building." [T] 

Limited resources 

"That is how the District could help us - ...provide substitute teachers so 
that you know that everyone can get the state wide training that PBS 
offers." [TM] 


Recommendations 

Specific recommendations can be drawn from the lists of contributors and barriers described above. In 
addition, some cross-cutting recommendations can be identified: 
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(1) Teacher commitment to the initiative needs to be developed and reinforced. Communication of 
guidelines and the rationale for practices could facilitate teacher buy-in. Systematic use of data 
to demonstrate the effectiveness of PBS practices was also linked to teacher buy-in. 

(2) Clear implementation guidelines should be provided to all school staff through a structured 
system of professional development and information dissemination. The guidelines need to help 
teachers respond positively to an often diverse population of students. They also need to 
encourage consistent and appropriate implementation of the rewards system based on a shared 
understanding of PBS in the school. 

(3) Systematic data collection and use allows teachers and administrators to monitor the nature, 
location, and frequency of smaller disciplinary infractions before they become larger issues. 

(4) Including student input in data-based decision-making can be useful in addressing barriers 
associated with the rewards system, including selecting appropriate rewards and monitoring the 
consistency with which rewards are offered. 

(5) PBS orientation/training for new and substitute teachers could increase the consistency of 
application within the school. Similarly, refresher training may be needed for more experienced 
teachers. However, given the limited resources for professional development and the 
experience of some teachers with PBS, creativity in how to provide the training - to whom, 
when, and in what format - will be needed. 


Analysis of the Relationship between Disciplinary Infractions and Schoolwide DE-PBS 

Methodology 

To answer evaluation question 2, differences in suspension rates over time in PBS and non-PBS schools 
were tested using data from 2005-2008, with separate models for elementary and middle schools. (Due 
to changes in how suspension data were reported to and collected by the state, data before 2005 or 
after 2008 were not used.) The sample was limited to elementary or middle schools that had 
participated in the student climate survey. In addition, PBS schools had to have participated in the PBS 
program for at least three years as of fall 2009. A total of 55 elementary schools fit the eligibility criteria, 
with 17 (31%) not implementing and 38 (69%) implementing PBS. Of the 18 middle schools that fit the 
eligibility criteria, four (22%) were not implementing and 14 (78%) were implementing PBS. Using 
suspension rate data from these schools, a longitudinal hierarchical linear modeling analysis (HIM) was 
used to account for the nested structure of the data (suspension rate data nested within schools over 
time). The original design planned to include other possible influences on discipline over time, such as 
prior academic performance, percentage of students with free and reduced lunch, school climate (as 
measured by The Delaware School Climate Survey - Student (DSCS-S), Bear, Smith, Blank, & Chen, in 
press), and level of PBS technical assistance (as measured by a survey of coaches). Although the sample 
included only those schools that had school climate survey data, not all schools agreed to share the data 
for research purposes. As a result, climate was not included as a variable in the analysis. Incomplete 
data on levels of technical assistance meant that this variable was also not included. All other school- 
level discipline, achievement, and demographic data were provided by the Delaware Department of 
Education. 

Findings 

In the analyses, neither PBS nor non-PBS schools showed significant change in the number of 
suspensions over time. That is, suspension rates were not significantly increasing or decreasing. Neither 



of these results are altered by including socioeconomic status or DSTP scores as covariates. In addition, 
no differences were found between PBS and non-PBS schools in 2005 suspension rates in either the 
elementary or middle school samples. 

The lack of evidence for a relationship between PBS and suspension rates can be attributed to six 
limitations: The first limitation was the inability to use an experimental design and randomly assign 
schools to PBS implementation or no implementation. Second, the use of suspensions as an outcome 
measure is problematic because of fluctuations that occur over time that are unrelated to student 
behavior and that they do not capture positive changes in student behavior. Third, the criteria for being 
included in the sample (such as implementing PBS for three or more years and having approval to use 
student climate survey data) reduced the size of the sample and thus the power to detect significant 
differences. Fourth, the covariates that we were unable to include in the analysis may have helped 
explain why PBS and non-PBS schools do not show significant differences in suspension rates. Fifth, 
schools in the PBS group had staggered implementation start points. Due to variation in how suspension 
data were collected and reported before 2005 and after 2008, we are unable to account for the 
difference in starting dates of implementation. Sixth, the qualitative data indicate that the category of 
"PBS school" (versus non-PBS school) is not clearly defined. The six case study schools are all PBS 
schools, but PBS was implemented in varying degrees and in varying ways in each school. At the same 
time, schools that are not implementing PBS may have programs that are similar in nature and intent to 
PBS. Thus, a quantitative comparison of PBS and non-PBS schools may have been conceptually flawed 
from the beginning. 

Recommendations 

Some of the issues that made these analyses problematic could be resolved with the following changes: 

(1) Sample size was a major limitation of this analysis. Studies involving more schools may provide 
more reliable estimates of direct effects. 

(2) The change in the kind of infraction data maintained by DDOE limited the ability to do 
longitudinal analyses over more than 3 years. Using more stable outcome measures would make 
it easier to examine how being a PBS school is related to intended outcomes over time. (For 
example, over time, does a PBS school have an improved school climate, or reach a threshold 
level of positive school climate?) 

(3) Analyses using the outcomes that are likely to be most directly affected by PBS practices might 
be more sensitive. Past research (Bear, 2010) has suggested that suspension data are highly 
related to office referrals. As noted previously, there are limitations to the use of suspension 
data. For example, the case studies here suggest that there is some manipulation of the 
suspension and office referral numbers by in-school practices discouraging referrals. Flowever, 
there may be other outcomes that are appropriate indicators of PBS effects. 

While these recommendations respond to some of the major flaws of our analyses, they do not address 
the more fundamental issue of the definition of "PBS" and what it means to be a PBS school in 
Delaware. 

Implications for Future Evaluations 

The findings from both the qualitative and quantitative components of the study point to next steps for 
evaluating DE-PBS. We know that among DE-PBS schools there are varying levels of implementation. 
Conversely, it is reasonable to assume that some schools that have not officially adopted PBS are 
implementing practices consistent with PBS approaches. If this is true, a more appropriate comparison 
would be between schools that implement practices consistent with PBS and those that do not. This 
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suggests two follow-up studies. The first would develop a rubric (or test of existing measures) to rate 
schools on how well it aligns with PBS principles. The second study would then build on the first with a 
quantitative analysis of effects on suspension rates (or other appropriate outcome measure) on 
randomly selected schools if possible, using the degree of fidelity with PBS principles - rather than 
whether the school was labeled as PBS or non-PBS - as the independent variable. In addition, the 
model could be used to explore whether additional school characteristics explain any differences in 
outcomes. 

Implications for DE-PBS 

The report closes with a postscript reflection from a PBS expert and member of the DE-PBS leadership 
team. The postscript discusses the evaluation results in light of specific program elements and 
programmatic efforts in Delaware, prior evaluation work conducted in Delaware, and includes a set of 
six recommendations for the future. The recommendations focus on specific steps the DE-PBS 
leadership team can take to facilitate improved program implementation through additional 
professional development, use of new instrumentation grounded in the Delaware PBS key features, and 
the strategic selection of schools for more intensive work. 
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Introduction 


In Spring 2010, the Delaware Education Research and Development Center conducted an evaluation of 
Delaware's PBS project, an initiative focused on developing a school-wide system of strategies to reduce 
behavior problems and foster a positive school climate. The study focused on facilitators and barriers to 
PBS implementation, and also included analyses to examine the relationship between PBS and school 
suspensions. This report first provides background on the principles of the original Positive Behavior 
Support program and the history of its implementation in Delaware, and then presents the evaluation 
questions, briefly outlines the methods used, and describes the results. The report closes with a 
postscript reflection on the findings from a PBS expert and member of the DE-PBS leadership team. 

PBS in Delaware: Background and History 

During the past decade, over 10,000 schools have adopted the Schoolwide Positive Behavioral Supports 
approach (SWPBS; Sugai & Horner, 2009) to schoolwide discipline (simply referred to as "PBS" in 
Delaware). This includes approximately 60% of public schools in Delaware. The rapid growth in the 
popularity of PBS can be attributed largely to schools seeking "positive" alternatives to a punishment- 
oriented, zero-tolerance approach to schoolwide discipline, but perhaps more so to federal funding 
provided in the 1997 and 2004 amendments to the Individuals with Disabilities Education Act (IDEA) for 
staff development and training in PBS. The purpose of such staff development and training is preventive 
- to reduce schoolwide behavior problems and the number of children referred to special education. 
This includes a focus on reducing significant discrepancies in suspension and expulsion rates between 
general and special education students, as well addressing significant disproportionality in identification 
and placement of minority students in special education, specific disability categories, and more 
restrictive settings. 

What is PBS? 

Schoolwide PBS is not defined in IDEA. However, most PBS programs, including most in Delaware, have 
followed the behavioral approach to school discipline advocated by George Sugai and Robert Horner at 
the University of Oregon. Firmly grounded in techniques of behavior modification and applied behavior 
analysis commonly used in programs for individuals with disabilities, this approach has five primary 
elements (Horner, Sugai, Todd, & Lewis-Palmer, 2005; Sugai & Horner, 2009): 

(1) research-validated behavioral practices, which consist primarily of the direct teaching of social 
skills via posting of 4-5 behavioral expectations (e.g., "Respect others," "Be responsible," "Do 
your best," etc.) throughout the school, rewarding students for good behavior (typically with 
tokens that can be exchanged for tangible rewards or privileges), and applying punitive 
consequences in a consistent fashion when students misbehave; 

(2) ongoing collection and use of data for decision making, with the collection and examination of 
office disciplinary referral data being most common; 

(3) student outcomes, with reduced office disciplinary referrals being the primary outcome 
measure in research studies; 

(4) supportive systems, including team-based implementation, use of district PBS coaches, 
administrative leadership, at least 80% documented commitment of school staff, an information 
system, adequate personnel and time, and budgeted support; and 

(5) a three-tiered model of prevention, interventions, and supports. Tier 1 targets all children and 
staff. Tier 2 targets small groups of children at-risk of exhibiting serious behavior problems 
and/or experiencing negative outcomes due to the presence of risk factors, and Tier 3 targets 
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individual students who exhibit chronic and serious behavior problems requiring intensive, 

comprehensive, and individualized interventions and services. 

Structure and History of PBS in Delaware 

Collaborative leadership. The DE-PBS initiative is a collaborative effort between the Delaware 
Department of Education (DDOE), the Center for Disabilities Studies at the University of Delaware, and 
Delaware public schools. The project is led by a PBS Core Team consisting of project staff members from 
the Center for Disabilities Studies (a project co-director and a research associate), the Delaware 
Department of Education (a DDOE staff member who serves as co-director). University of Delaware 
faculty partners (two professors in the School of Education who serve as consultants), and a consulting 
school psychologist who works in the public schools. In previous years, resident teachers with the DDOE 
have also served on the Core Team. 

Changes over the years. Although DE-PBS has always adhered to the five basic elements of 
SWPBS listed above, its focus has shifted over the years. During the first few years of the project (1999- 
2001), nearly all resources were devoted to developing training modules for schools on PBS practices at 
all three tiers, with an emphasis on supports and interventions for individual students (Tiers 2 and 3). In 
2001, the focus shifted to preventing behavior problems at the schoolwide level (Tier 1) - this focus 
remains today, although staff development and training specific to Tiers 2 and 3 are also provided. Staff 
development and training workshops were developed and provided statewide to assist schools in the 
implementation of the five basic elements of PBS described above. Since 2002, these statewide 
workshops have been refined and new ones have been added. 

In addition to the training workshops, the project provides technical assistance to a cadre of district PBS 
coaches. PBS coaches, who have a portion of their time assigned to implementing PBS in their districts, 
play a critical role in the implementation of PBS at the building level. The PBS Cadre meets regularly 
throughout the school year, during which specialized trainings are provided. The Cadre also serves in an 
advisory capacity to the Core Team. 

Until recently, increasing the number of schools implementing PBS has been a primary goal of the 
State's PBS initiative. As shown in Figure 1, the project has been successful in achieving this goal. With 
the rapid increase in the number of PBS schools has come great variability across PBS schools in 
important outcomes, including school climate, office disciplinary referrals, school suspension rates, and 
academic achievement scores. In a preliminary evaluation of DE-PBS (Bear, Blank, & Pell, 2009), results 
suggested no significant differences between PBS and non-PBS schools on the above outcomes. 
However, conclusive results could not be obtained because of the lack of randomized assignment to PBS 
and non-PBS schools and the likelihood that any differences, or lack thereof, between PBS and non-PBS 
schools could at least partially be explained by differences between schools in variables not measured 
(e.g., it appeared that PBS schools, compared to non-PBS schools, were more likely to be urban and to 
have a greater number of students at risk for behavior problems). Although not conclusive, these results 
sparked a new direction in the DE-PBS initiative during the 2007-2008 school year which continues 
today. 
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Figure 1. Number and percentage of schools implementing PBS by year in DE 
A New Direction: Current Status of DE-PBS 

Although many more schools are now designated as PBS schools, the rapid increase was 
associated with variability in the quality of implementation of the program across schools. To address 
the implementation issues, and increase the likelihood that the program will have the desired effects, 
the project has focused on three critical tasks in recent years. These tasks are: (1) developing key 
features of DE-PBS, including the valued student outcomes and the processes for obtaining them, (2) 
creating reliable and valid evaluation measures of those outcomes and processes, and (3) aligning key 
features and assessment measures with staff development and training. 

Current Key Features of DE-PBS. 

During the 2007-2009 school years, a workgroup comprised of PBS Core Team members and district 
coaches developed 10 key features of PBS in Delaware to guide program development, evaluation, and 
staff development. In Delaware, a PBS school is supposed to: 

(1) Recognize that a positive and safe school climate promotes not only positive behavior, but also 
academic, social, and emotional development. 

(2) Recognize that ALL students benefit from positive behavioral supports. 

(3) Recognize the critical importance of preventing behavior problems. 

(4) Recognize the critical importance of developing self-discipline. 

(5) Recognize the critical importance of correcting misbehavior using a combination of evidence- 
based techniques for increasing appropriate behavior and decreasing use of inappropriate 
techniques. 
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(6) Recognize the critical importance of providing students who exhibit serious or chronic behavior 
problems with comprehensive and intensive evidence-based interventions and supports. 

(7) In translating these beliefs into practice, adopt a problem-solving team process for planning, 
development, implementation, and evaluation of evidence-based practices across all three 
levels of prevention and intervention (primary, secondary, and tertiary). 

(8) In translating these beliefs into practice, demonstrate sustained commitment, participation, and 
implementation with fidelity by the majority of staff, administrators, district leadership, and 
school community in a shared approach to the dynamic and evolving PBS process. 

(9) In translating these beliefs into practice, value the importance of data-based decision making, as 
reflected in the on-going evaluation of program effectiveness and modification of program 
components, interventions and supports based on multiple sources of data. 

(10) In translating these beliefs into practice, provide on-going professional development and 
support to school staff that corresponds closely with the needs of the schools and individual 
staff members. 

In addition to key features, an overall vision, or project aim, was established: to create safe and caring 
learning environments that promote the social-emotional and academic development of all children. The 
vision statement and key features depart from the project's previous focus on the application of 
behavioral techniques to change student behavior, as found in the Sugai and Horner approach. Instead, 
greater emphasis is now placed on the use of a variety of evidence-based techniques and strategies not 
only for preventing and correcting behavior problems at the schoolwide and individual student levels, 
but also for developing self-discipline and fostering a positive school climate. Such techniques include 
behavioral techniques, but also techniques more commonly associated with social and emotional 
learning (SEL) programs designed to promote social and emotional development. A substantial body of 
research supports those programs (Durlak et al, 2010) and there is growing recognition among 
researchers that a combination of PBS and SEL strategies and techniques are necessary for effective 
school discipline (Bear, 2010; Osher, Bear, Sprague, & Doyle, 2010). Delaware appears to be one of the 
first states to attempt to combine PBS and SEL techniques. 

Valid and Reliable Measures of Outcomes and Processes 

Guided by the key features, the project has devoted increased efforts and resources to the development 
of three measures: the Delaware School Climate Surveys (DE-SCS; Student, Teacher/Staff, and Home 
versions), the Delaware Assessment of Strengths and Needs for Positive Behavior Supports (DE-ASNPBS) 
and the Delaware Evaluation of Key Features (tentative title; DE-EKF). Supported by additional funding 
by the DDOE, over the past 5 years the DE-SCS has been administered to PBS and non-PBS schools in 
Delaware, including 155 schools during this past spring. Evidence of the DE-SCS's reliability and validity 
are firmly established. The three versions of the DE-SCS appear in a recent book (Bear, 2010) and the 
student version will be published this year in the Journal of School Psychology (Bear, Smith, Blank, and 
Chen, in press). The DE-ASNPBS, consisting of 50 items based on research on classroom management 
and school effectiveness, was completed this past fall and recently administered to schools this spring. 
Completed by teachers and staff of each school, this instrument is intended to be used as a process 
evaluation tool, as well as to help the project identify techniques and strategies that might differentiate 
effective and ineffective schools with respect to schoolwide discipline. Finally, the project is currently 
developing the DE-EKF a tool to be completed by outside evaluators (i.e., staff from the Center for 
Disabilities Studies or the DDOE) during a site visit to evaluate fidelity of PBS implementation. It is 
expected that schools that implement the key DE-PBS features with the greatest fidelity will be the most 
effective schools. 
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In assessing the effectiveness of PBS and the process of its implementation, the project intends to use 
each of the above measures in combination with office disciplinary referral data and school 
suspension/expulsion data. This will require greater attention to the reliability and validity of the latter 
measures. 

Alignment of Key Features and Measures with Staff Development and Training 

With completion of the key features and near completion of the assessment measures, efforts are now 
underway to ensure that the current staff development and training opportunities are well aligned with 
the key features and assessment measures and are provided in sufficient depth and scope. 

In sum, DE-PBS is currently in a state of transition - evolving from a primarily behaviorally-oriented 
approach to managing student behavior to a more comprehensive approach to schoolwide discipline 
designed not only to help manage student behavior, but also to develop self-discipline and foster 
positive school climate. The success of this transition depends on the development and provision of 
adequate staff development and training opportunities, as well as on the development of reliable and 
valid outcome and process measures aligned with the project's key features. These are the current focus 
of DE-PBS. To assist in these efforts, project staff are currently writing a $4 million dollar grant proposal 
to the Office of Safe and Drug-Free Schools 

Evaluation Questions 

As the DE-PBS Core Team worked on the tasks of identifying key features, developing new tools, and 
refocusing their staff development activities, two evaluation questions emerged. The first is formative, 
intended to help the team in its current transition. The second looks again at the outcome of disciplinary 
infractions, using a different analysis than that used by Bear and colleagues (2009) to see if the results 
are any different. The specific questions are: 

(1) What are contributors and barriers to schools' implementation of DE-PBS? 

(2) What is the relationship between DE-PBS and changes in incidence of disciplinary infractions? 
These questions were developed in consultation with members of Delaware's PBS state-level team. 

The next two sections present the methods and findings for each question, respectively. As will be 
described, a qualitative multiple case study design was used for the first question. The second question 
was addressed with hierarchical linear modeling, using suspension data from 2005-2008, provided by 
the Delaware Department of Education. Each section ends with a brief set of recommendations, which 
are then integrated into a discussion of future directions. To put the recommendations and future 
directions in the context of previous PBS research and the experience of PBS in Delaware, the report 
ends with a reflective postscript from a member of the DE-PBS Core Team who has played a primary role 
in the research aspects of the program's implementation. It includes another set of recommendations 
that reflect this perspective. 


Contributors and Barriers to DE-PBS Implementation 


Methodology 

The study used a multiple case study design to learn about contributors and barriers to PBS 
implementation. Only those schools that had been implementing PBS for at least three years were 
included in the list of potential cases. From that group, elementary and middle schools were purposively 
sampled to bracket the highs and lows of school disciplinary infractions, as measured by the suspensions 
and expulsions reported to DDOE. At each level (elementary and middle), three schools exhibiting high 
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levels of disciplinary infractions and three schools exhibiting low levels of disciplinary infractions were 
selected, resulting in a total of 12 schools as potential participants. While the focus of the sampling 
strategy was on representing the highs and lows of disciplinary infractions, efforts were also made to 
obtain diversity across counties and school districts. 

The twelve schools were distributed across six districts. The district superintendents were contacted 
asking permission to include these schools in the study. Of the six districts contacted, four supported 
the participation of the schools selected. Schools for which district leaders granted permission were 
then contacted directly to see if they were willing to participate. Of the original 12 schools identified for 
potential participation, seven were contacted directly and six chose to participate. The final sample 
includes three middle schools and three elementary schools, at least one school from each county, and 
two schools with a high rate of disciplinary infractions and four with low rates. The participating schools 
are not identified in this report because the focus is on the cross-cutting findings of barriers and 
contributors, rather than on successes or challenges at specific schools. 

In the six selected schools, the administrators serving on the PBS team were asked to participate in 
individual interviews. All members of a school's PBS team were invited to participate in a focus group 
interview. Focus groups were also held with teachers and, in the schools with low rates of disciplinary 
infractions, with students. (Given time constraints, focus groups could not be held with students at all 
schools.) In all but one school, participants in the teacher and student focus groups were randomly 
selected from lists provided by the schools. (One principal did not want to provide us with teacher and 
student lists, and instead told us that he/she identified participants by following our random selection 
procedures.) A maximum of 10 participants were included from those returning completed individual 
(and, for students, parental) consent forms. District PBS coaches for each school were also interviewed. 
For details about the interview data collection and analysis procedures, see appendix A. The interview 
questions are in appendix B. 

With the exception of the students, all school interviewees were asked to complete an online version of 
the "Delaware Assessment of Strengths and Needs for Positive Behavior Supports" (Bear, Burwell, Baker, 
Blank, & Boyer, 2009).^ With items aligned with the key features of DE-PBS and drawn from the research 
literature on school discipline and classroom management, the instrument is designed to help schools 
assess and examine their strengths and needs related to schoolwide and classroom discipline efforts. 
Using 5-point scales, respondents are asked to rate the importance of items and indicate the extent to 
which each item is a strength or weakness at the school and classroom levels. In this study, respondents 
from schools with high incidence of disciplinary infractions and those from schools with low incidence 
gave similar ratings of the importance of the items. Flowever, confirming the diversity of the selected 
schools - and thus their likely utility in providing different perspectives on DE-PBS, the survey results 
show that respondents from schools with higher incidence of disciplinary infractions gave lower ratings 
on the strength of components than those from schools with lower incidence of infractions. For 
example, average ratings across components on schoolwide discipline were 3.3 for the high-incidence 
schools and 4.2 for the low-incidence schools. Similarly, the average ratings for classroom discipline 
were 3.5 and 4.3 for high and low-incidence schools, respectively. More details about the response 
rates, participants, and average ratings for each component for the two sets of schools can be found in 
appendix C. 


^ PBS team members and teachers received a $75 debit card as an incentive to participate in the interview and 
complete the survey. Neither coaches (who did not complete the survey) nor administrators received incentives. 
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The data from the individual and focus group interviews were transcribed from audio recordings and 
then analyzed using a combination of deductive categories drawn from the key features of DE-PBS and 
inductive coding to identify emergent themes. In keeping with the focus on cross-cutting contributors 
and barriers rather than on the experiences of individual schools, the analysis was conducted by type of 
respondent first (that is, all PBS team focus groups, then all administrators, and so on) and then the 
results were aggregated across the different respondents to identify major categories and specific 
contributors and barriers. This approach limits the opportunity to include the role of school context in 
the analysis. The goal of this part of the evaluation was to help the state PBS team learn how to improve 
school's implementation of PBS, whatever its context. 

Findings 

This study found four dominant themes related to contributors and barriers to the implementation of 
DE-PBS: 

(1) consistency and adaptability, 

(2) rewards, 

(3) data-based decision making, and 

(4) professional development and support. 

These themes encompass both contributors and barriers to PBS implementation. For example, the 
theme of consistency and adaptability includes the positive aspect of PBS evolving over time in response 
to changing needs, and the concern about PBS becoming stagnant and unresponsive. Each theme is 
discussed below, followed by a special section summarizing student perspectives. Illustrative quotes 
from the interviews accompany each theme, with additional quotes in appendix E. 

Theme 1: Consistency and Adaptability 

Those rules are throughout the whole school, not only in the classroom, but it's being safe in the 
hallway, in the playground, on the bus, and In the cafeteria. It starts in first grade and it goes all 
the way through 5^^ grade. The rules are the same no matter where you are and no matter what 
grade you are In. (Teacher) 

There are still considerable people, teachers and staff who are resistant and that clearly affects 
how well they are able to Implement. There are some naysayers that clearly don't want to be on 
board. (Coach) 

Consistency and adaptability, the most prominent factor identified in the data, has three dimensions: 
the consistency of commitment to PBS over time; the consistency with which school-specific PBS policies 
and practices were applied within each school; and the adaptability of PBS as part of a dynamic and 
evolving school culture. (See Table 7 in appendix E for quotations illustrating the features.) 

Contributors 

Consistent commitment to PBS. Consistent commitment is manifested in teacher and student 
buy-in. Teachers who indicated a lack of buy-in expressed a sense that PBS was imposed on them from 
the outside or from the top down. In contrast, those who felt a commitment to PBS described it as a 
positive and supportive part of their everyday practice. In many cases they also expressed the belief that 
PBS was something they were co-constructing within the school community. 

Consistency of policies and practices. Several respondents emphasized the importance of 
consistent internal practices. Teachers, administrators, PBS team members, and coaches all spoke of this 
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as necessary for the successful implementation of PBS. When practices were perceived as being 
implemented uniformly across classrooms and other school sites, teachers expressed confidence in their 
ability to respond appropriately to behavioral issues. Under these circumstances teachers expressed the 
belief that they were operating as part of a team. They felt it was easy to communicate clearly to 
students because all adults were using the same terms to convey expectations. Students also expressed 
satisfaction with consistency of application. They suggested that PBS helped them to remember how to 
act and gave them a tool to remind their peers of the behavioral expectations in their school. 

Adaptability, innovation, and experimentation. In addition to the consistent application of PBS 
principles and practices, a willingness to "change things up" as necessary to avoid stagnation was 
identified as an important contributor to PBS implementation. The key to successfully balancing these 
two seemingly contradictory drives appears to be maintaining and sustaining open lines of 
communication within the school. In other words, when new practices are introduced, they are 
supported by the reasons for the change, clear guidance on their implementation, and sufficient 
resources. Information is required to learn when adaptation is necessary. 

Another form of adaptability is the use of PBS in combination with other programs. In some cases, these 
combinations seemed to indicate a view that PBS can be shaped to meet the needs of the school. 
However, in other cases, participants described PBS as an ad hoc tool to be pulled out of the tool bag as 
necessary, which could be interpreted as an incomplete, as opposed to flexible, implementation of PBS. 

Barriers 

Barriers to achieving the principles of consistency and adaptability included lack of buy-in to DE-PBS 
philosophy in general or to DE-PBS as implemented in the school, and a lack of understanding of DE-PBS 
implementation guidelines. 

Lack of buy-in. A lack of commitment to the principles of PBS was expressed particularly in the 
context of the rewards system (see more about rewards below). For example, some respondents 
expressed disdain for practices that they believed reward students for behaviors that should be 
standard expectations. They described the PBS program as being based on a form of bribery. In some 
cases, they viewed the students as manipulating the system to maximize their rewards. This resistance 
to PBS principles was often accompanied by reports of individual teachers altering PBS practices to fit 
their personal beliefs about classroom management. 

Limited understanding of guidelines. Some teachers reported that they were not provided with 
adequate guidelines to implement PBS in their classrooms. This led to different teachers communicating 
different expectations to students. When these inconsistencies became apparent to students, they were 
able to identify which teachers were likely to issue rewards and which teachers were not. The lack of 
buy-in from the teachers seems to result in a lack of buy-in from the students and was therefore a 
barrier to PBS implementation. 

Theme 2: Rewards 

With the school store it has changed because of the survey that was taken by fifth grade 

students and they came up with different things they would like to see in the school store. 

(Teacher) 

Like in middle school they're not going to want to be winning candy, they're going to want to win 

money and stuff that they can actually use. (Student) 
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When asked to describe the major components of PBS in their school, almost all respondents mentioned 
the rewards system before any other element of the program. The structure of rewards systems was 
similar across the schools. Each school had a currency system in place wherein students were issued 
varying amounts of the school currency for different types of positive behavior. Although the currency 
was a common element, there was variation in how it was used. In different schools, students could use 
the currency to buy pre-priced items in a PBS store, bid on items at auctions, participate in a weekly 
lottery, or purchase extra privileges. In some cases, schools used a combination of group activities and 
events with individual prizes. In the six schools that we studied, rewards seem to dominate the 
definition of PBS. 

While the structure was similar across sites, the perceptions of and beliefs about the value of the 
rewards system as expressed by members of the school communities varied greatly. In some instances 
rewards were viewed as invaluable contributors to the success of the PBS program, but in others there 
were concerns about how the rewards system was being implemented. (See Table 8 in appendix E for 
some quotations illustrating the pros and cons of the rewards systems.) 

Contributors 

Student input. We heard descriptions of carnivals, pep rallies, fun days, and field trips from 
which students could participate or be excluded based on their discipline record. Where the events were 
well planned and where students were polled for their input, these rewards had the potential to be 
powerful incentives for good behavior. Students also expressed positive interest in attending these 
events and were described by adults as the best ambassadors for PBS based activities. Some schools 
built an element of choice into this aspect of their rewards model in which students were presented 
with five or six potential events from which they could select one. All events that had enough student 
interest were run simultaneously. 

Nature and timing of rewards. Some participants suggested that more immediate rewards 
provided greater incentive for positive behavior and that some students were less likely to be swayed by 
an event that would take place in two weeks or two months time. Some rewards were symbolic rather 
than the currency or activity incentives. For example, in one school, student teams competed for a token 
trophy that was awarded on the basis of cumulative group behavior. In another school dress down days 
were seen as a significant incentive. 

External support. Some respondents said that PBS stores were desirable because they offered 
the opportunity to use the school currency immediately, without waiting for an event. However, the 
stores require resources that not all schools had. PBS teams and administrators generated external 
support for PBS stores and other elements of the program in a number of ways. Some schools sought 
support and donations from the local community. Community businesses were sometimes invited to the 
larger PBS events to demonstrate the value of their contributions and to foster on-going support for a 
school's initiatives. Writing proposals for grants from corporations was another activity to generate 
support. 

Barriers 

The reward systems were central to PBS implementation in the schools we studied, but the potential for 
corruption or misuse of the rewards system was viewed as a significant barrier to the success of PBS in a 
school. 

Lack of clarity/agreement about when to use rewards. Many teachers expressed the concern 
that the rewards, especially the currency, and not the positive behaviors that prompted the rewards, 
had become the primary motivating factor in students' actions. These teachers also expressed concern 
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about feeling pushed to reward students for behaviors that they felt should be considered standard. 
Inconsistent implementation was thought to damage buy-in by students. An administrator and 
numerous PBS team members described having to repeatedly remind some teachers to issue currency 
as a regular part of classroom practice, not as an occasional special event. Some teachers expressed the 
desire for more and clearer guidelines about what behaviors should be rewarded and with how much 
currency. 

Student manipulation of the system. Some teachers felt that varying practices among their 
colleagues meant that students were more likely to behave positively for teachers with a habit of over- 
paying. In addition middle school students talked openly about the differences in teacher use of the 
rewards, identifying which teachers used a lot of currency and which teachers would take the rewards 
back as a punishment. In such cases the inconsistent rewards seemed to drive selectively positive 
behaviors, rather than consistent rewards driving consistently positive behaviors. 

Rewards that are inappropriate for the age group. Another barrier related to the rewards 
system is the issue of age appropriateness. Almost all schools described students losing interest in PBS 
rewards as they advanced through the grade levels. This was particularly a problem at the middle school 
level where graders, and some graders, were no longer eager to participate in the PBS rewards 
system. Teachers described issuing currency to students who simply left the tickets behind in class or 
threw them in the trash. While some schools were attempting to differentiate rewards on the basis of 
age appropriateness, insufficient resources was reported to be a significant challenge to doing this 
consistently across time. 

Theme 3: Data-Based Decision Making 

We look at where they [infractions] are occurring, are there repeat offenders, and what have we 
done for those repeat offenders. Are they seeing the guidance counselor, so they have an FBA 
(Functional Behavior Assessment), do they have an lEP that has accommodations. We try to look 
at where things are occurring, who they are occurring with and then what can we do about that. 
(Administrator) 

...there was nothing in any of the monthly meetings that talked about data. It was more like 
what are we going to do for celebrations. (Coach) 

Data-based decision-making is a key feature of PBS and was addressed specifically in our interview 
questions. As described below, the respondents indicated the positive aspects of using data, but some 
barriers to the use of data were also identified. (See Table 9 in appendix E for some verbatim comments 
on the subject of data-based decision-making.) 

Contributors 

Many school and district staff in our interviews agreed that the best way to monitor and refine PBS 
programs was to systematically collect and use data on the program and students. Feedback from 
respondents indicated that using data in decision-making facilitated PBS implementation in three main 
ways: allowing school staff to respond to emerging issues quickly; making it easy to monitor and react to 
individual student behavior, and building staff buy-in. 

Quick response to emerging issues. Some administrators described how access to data enabled 
them to respond quickly as problems came up. For example, at one school the PBS team described using 
data to identify a potential problem in the cafeteria area before it became a larger issue. They then 
formulated a plan that altered current PBS practices to address the emergent need. In another case, an 
administrator demonstrated how the school's data system allowed the monitoring of current activities 
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in the school related to discipline and how that information could be compiled over time to identify 
potential trends. 

Monitoring individual behavior. While data were used to keep track of disciplinary trends in the 
school overall, data were also important in monitoring the behavior of individual students. One 
respondent described how they used the individual level information to identify students whose 
behavior had improved over time so that the students' progress could be recognized and communicated 
to the parents. In another case, the information was used to identify "repeat offenders" who were then 
followed up to see if they had seen the guidance counselor or had other kinds of assessments and 
interventions to help them manage their behavior. 

Increased staff buy-in. Decisions based on sound data were also reported to promote teacher 
buy-in, or at the least reduce the doubts of perceived "naysayers". As some participants noted, simply 
accumulating data is not sufficient. There must also be the willingness and capability to appropriately 
analyze and use the data in decision-making. Some respondents valued having regular check-ins. Teams 
that communicated awareness of the need to continuously review the program for strengths and 
weaknesses also appeared to have a strong sense of internal efficacy in terms of their ability to adapt or 
discard current practices. 

Barriers 

While data itself is not perceived as a barrier to successful PBS implementation in a school, data for 
data's sake and a failure to support practice with evidence of effectiveness can lead to a loss of faith in 
any program over time. 

Data as window-dressing. Some respondents described the compilation of data that did not 
result in any subsequent action. Teachers in some schools were unsure as to what was being tracked 
and how. They expressed a feeling of being "out of the loop" when it came to disciplinary data. In 
contrast, other schools produced monthly reports and presented disciplinary data, its potential 
implications, and their proposed alterations to practice at general staff meetings. 

Unsystematic collection of data. PBS district coaches were particularly aware of the successes 
and shortcomings of different schools when it came to using data. They suggested that some schools 
had not implemented a rigorous system of data collection and appeared to be relying solely on the 
assessment of isolated individual incidents for the formulation of policy and practice. In such schools, it 
was not clear if disciplinary actions that did not rise to the level of in-school or out-of-school suspensions 
were being monitored. 

Professional Development and Support 

I would like professional development for my staff I would like them to see PBS schools. Like I 
wish there was a way that I could take a chunk of staff at a time and go somewhere to see how It 
works. And then I would like to provide incentives not only for my kids but for my 
staff. (Administrator) 

We did offer it [professional development]. ..but there are staff that have been here for a long 
time that didn't go through the cluster. There are some folks out there that are still not exactly 
sure of the basic philosophy. (PBS team member) 

Participants' perceptions of and beliefs about the role of professional development as a contributor to 
the success of PBS in their schools emerged as a significant factor across all adult focus groups and 
interviews. The theme of professional development and support was expressed primarily as a gap that 
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created a barrier to successful PBS implementation; however some facilitators were identified. (See 
Table 10 in appendix E for some respondent comments about professional development.) 

Contributors 

Awareness of district coach as a resource. While many schools do not appear to utilize the 
coaching services offered by the district, most were aware that the district coach was available if they 
needed additional support. Despite the fact that teachers in general felt there would be benefits to on- 
going professional development and refresher training, some coaches and PBS team members 
suggested that PBS had become so embedded within the culture of the school that it was essentially 
running itself. 

Annual refreshers. Some schools appear to be engaging in on-site PBS refreshers systematically 
at the beginning of each school year and at intervals throughout the year. They described PowerPoint 
presentations, lesson plan templates, and modeling that was used to remind teachers of the basic 
philosophy and practices of PBS in their schools. As an alternative to orientation training, some schools 
had compiled a PBS orientation binder that was to be made available to all new teachers in the school. 

Barriers 

Lack of training or lapses in on-going professional development were identified as a barrier to successful 
PBS implementation. While most schools appear to have received some form of professional 
development for teachers at the outset of PBS implementation, insufficient training for new teachers, 
lack of refresher training for experienced teachers, and limited resources for professional development 
were specific concerns. 

Insufficient training for new teachers. Some teachers stated that they had never received any 
form of training in PBS. These teachers were, for the most part, relatively new to the school and did not 
receive PBS orientation training prior to commencing employment. One such teacher described 
spending his first three weeks in his classroom without any knowledge of what he was supposed to do 
with the PBS tickets. 

Lack of refresher training. In most cases schools had provided limited refresher training for 
long-term teachers. One teacher suggested that the inconsistencies in PBS practices at her school could 
be addressed by providing teachers with further professional development. Some teachers who had 
been in their current building for a number of years could not recall ever receiving formal professional 
development in PBS, but acknowledged that PBS information was disseminated among the staff by the 
PBS team. 

Limited resources for professional development. While most of the administrators interviewed 
would like to have more opportunities available for staff training, they acknowledged that time and 
resources are often a barrier to doing so. Teachers also identified the lack of time or support for 
professional development as a challenge. Some administrators reported relying on the PBS team for the 
dissemination of pertinent information among teachers and other adults in the school. In some cases 
PBS training consisted primarily of PBS team presentations at general staff meetings. 

Students' Perspectives 

I would say that most teachers don't really realize that some kids are doing something good. I would 
get teachers to notice more and to point kids out to say they're doing a good job and stuff. (Student) 

Some of the teachers are strict, they use PBS against us. Like they'll take away tickets. Some teachers 
don't use tickets, or they'll give them if you pass a test. One teacher will give them if you sharpen 
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your pencil [said with a mocking tone]! The bus driver can give you tickets for behaving on the bus. 
(Student) 

As part of our data collection we conducted student focus groups in four of the six schools. The content 
of these focus groups was somewhat different from the other interviews we conducted. However, one 
similarity is that the students also spoke primarily about PBS as consisting of the rewards system. They 
demonstrated a clear grasp of how the rewards system works and what types of behavior merited 
rewards. There seemed to be a consensus that larger PBS events were more attractive to many students 
than smaller rewards. They seemed to be aware that attendance at these events was determined by 
their behavior over time and that they were being monitored based on in-school and out of school 
suspensions. 

Underlining the theme of consistency in general, and rewards in particular, students also spoke about 
inconsistent PBS practices. They described certain teachers handing out many rewards tickets for 
ordinary behaviors, while others rarely if ever issued rewards tickets. In some cases students joked 
about opening the door or cleaning the board for certain teachers who were known to issue many 
reward tickets. While there were many comments about the inconsistency with which rewards were 
distributed, some students also indicated that teachers are oblivious to the positive behaviors that 
students engage in and recommended that they take more time to acknowledge good behavior. 

When asked about improving PBS, students across all sites suggested that their school needed to 
implement harsher consequences for bad behavior. They felt that lenient discipline meant that the same 
students continued to get into trouble repeatedly and that current consequences were not serious 
enough to deter them from doing so. (See additional quotes from the student focus groups in Table 11 
in appendix E.) 

Recommendations 

The issue of consistency emerged across the six case studies as central to implementation. Consistency 
was addressed specifically, and it also shows up in the other themes. In the discussion of the rewards 
theme, the inconsistent use of rewards by teachers within a school was identified as a concern. Across 
schools, inconsistent attention to the key feature of data-based decision making was mentioned by the 
district coaches. Professional development and support also vary in ways that either support or 
undermine consistency. As noted above, some tailoring and flexibility is appropriate, but it raises the 
question of how much adaptation is acceptable before the school can no longer be considered to be 
implementing PBS? 

Specific recommendations can be drawn from the lists of contributors and barriers described above. In 
addition, some cross-cutting recommendations can be identified: 

(1) Teacher commitment to the initiative needs to be developed and reinforced. Communication of 
guidelines and the rationale for practices could facilitate teacher buy-in. Systematic use of data 
to demonstrate the effectiveness of PBS practices was also linked to teacher buy-in. 

(2) Clear implementation guidelines should be provided to all school staff through a structured 
system of professional development and information dissemination. The guidelines need to help 
teachers respond positively to an often diverse population of students. They also need to 
encourage consistent implementation of the rewards system based on a shared understanding 
of PBS in the school. 

(3) Systematic data collection and use allows teachers and administrators to monitor the nature, 
location, and frequency of smaller disciplinary infractions before they become larger issues. 
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(4) Including student input in data-based decision-making can be useful in addressing barriers 
associated with the rewards system, including selecting appropriate rewards and monitoring the 
consistency with which rewards are offered. 

(5) PBS orientation/training for new and substitute teachers could increase the consistency of 
application within the school. Similarly, refresher training may be needed for more experienced 
teachers. However, given the limited resources for professional development and the 
experience of some teachers with PBS, creativity in how to provide the training - to whom, 
when, and in what format - will be needed. 


Analysis of the Relationship between Disciplinary Infractions and Schoolwide DE-PBS 
Methodology 

To answer evaluation question 2, differences in suspension rates over time in PBS and non-PBS schools 
were tested using data from 2005-2008, with separate models for elementary and middle schools.^ The 
sample was limited to elementary or middle schools that had participated in the student climate survey. 
In addition, PBS schools had to have participated in the PBS program for at least three years as of fall 
2009. Table 5 shows the distribution of the sample of eligible schools across school level and PBS and 
non-PBS designations. 


Table 5. Distribution of eligible schools by school level and PBS designation 


Level 

PBS 

Non-PBS 

Total 

Elementary 

38 (69%) 

17 (31%) 

55 (100%) 

Middle 

14 (78%) 

4 (22%) 

18 (100% 


The sample of elementary schools included schools from all three counties and 12 school districts. For 
middle schools, PBS and non-PBS schools came from New Castle and Sussex Counties, while Kent County 
supplied only non-PBS schools. Ten school districts were represented among both PBS and non-PBS 
middle schools. 

To account for the nested structure of the data (suspension rate data nested within schools over time), a 
longitudinal hierarchical linear modeling analysis (HIM) was used. The original design planned to include 
other possible influences on discipline over time, such as school climate^ level of PBS technical 
assistance^, prior academic performance, and percentage of students with free and reduced lunch. 


^ Due to changes in how suspension data were reported to and collected by the state, data before 2005 or after 
2008 were not used. 

^The Delaware School Climate Survey - Student (DSCS-S) (Bear, Smith, Blank, & Chen, in press) measures student 
perceptions of school climate on a set of six factors: Teacher-Student Relations, Student-Student Relations, 
Fairness of Rules, School Safety, School Problems, and Liking of School. 

The PBS Technical Assistance Survey was developed in collaboration with PBS Core Team members to capture the 
level of technical assistance (TA) provided to schools by district PBS coaches. Coaches were asked to rate each PBS 
school from their district selected for our sample on the level of TA they received during the first two years of 
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However, although the sample included only those schools that had school climate survey data, not all 
schools agreed to share the data for research purposes. As a result, climate was not included as a 
variable in the analysis. In addition, incomplete data on levels of technical assistance meant that this 
variable was also not included. School-level discipline, achievement, and demographic data were 
provided by the Delaware Department of Education and used in the analysis. More details on the 
modeling performed can be found in appendix F. 

Findings 

In the analyses described above, neither PBS nor non-PBS schools show significant change in the number 
of suspensions over time. That is, suspension rates have not been significantly increasing or decreasing. 
Neither of these results is altered by including socioeconomic status or DSTP scores as covariates. In 
addition, no differences were found between PBS and non-PBS schools in 2005 suspension rates in 
either the elementary or middle school samples. More detailed results are presented in appendix F. 

Limitations 

The lack of evidence supporting a relationship between PBS implementation and suspension rates can 
be linked to design, analytical and conceptual weaknesses. First, the inability to employ a true 
experimental design in which schools were randomly assigned to PBS and non-PBS conditions, or to 
control for variables that might largely account for any differences in suspension rates between PBS and 
non-PBS schools (e.g., percentage of students at-risk for behavior problems) contribute to weaknesses in 
the study's design. Second, there are serious limitations to the use of suspensions, as well as office 
disciplinary referrals, as an outcome variable (Bear, 2010). For example, fluctuations of 30% and more in 
suspension rates from year to year are not uncommon in many schools, especially when there are 
changes in administration and staff (Wright & Dusek, 1998). Another limitation to suspensions as an 
outcome variable is that they do not necessarily reflect any positive changes in student behavior. That is, 
suspensions might decrease as a function of changes in teacher/staff behavior and not necessarily 
reflect improvements in student behavior or school climate. 

There are three analytical issues that may contribute to the lack of evidence supporting a relationship 
between PBS implementation and suspension rates. First, while limiting the analysis to schools 
implementing PBS for three or more years that also had student climate survey data, and separating the 
analysis between middle and elementary cases, made theoretical sense, those restrictions reduced the 
size of the sample and thus the power to detect significant differences. Second, the co-variates that we 
were unable to include in the analysis may have helped explain why PBS and non-PBS schools do not 
show significant differences in suspension rates. This possibility is reinforced by the significant amount 
of variance left in the model to be accounted for. Third, schools in the PBS group had staggered 
implementation start points. Due to variation in how suspension data was collected and reported before 
2005 and after 2008, we are unable to account for the difference in starting dates of implementation. 

Viewed in light of the qualitative case studies, the quantitative analyses have a more fundamental 
limitation than that posed by sample size and covariates. Specifically, the qualitative data indicate that 
the category of "PBS school" (versus non-PBS school) is not clearly defined. The six case study schools 
are all PBS schools, but PBS was implemented in varying degrees and in varying ways in each school. At 
the same time, schools that are not implementing PBS may have programs that are similar in nature and 


implementation and also on the maintenance years beyond the initial two. Coaches were also asked to comment 
on their ratings to help the researchers better understand the school ratings. The survey was administered online 
using SurveyMonkey and each coach was provided a unique survey containing only the schools she needed to 
rate. (See appendix D for the Technical Assistance Survey.) 
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intent to PBS. Thus, a quantitative comparison of PBS and non-PBS schools may have been conceptually 
flawed from the beginning. As discussed in the final section of the report and the postscript, this 
suggests some direction for future research. 

Recommendations 

Some of the issues that made these analyses problematic could be resolved with the following changes: 

(1) Sample size was a major limitation of this analysis. Studies involving more schools may provide 
more reliable estimates of direct effects. 

(2) The change in the kind of infraction data maintained by DDOE limited the ability to do 
longitudinal analyses over more than 3 years. Using more stable outcome measures would make 
it easier to examine how being a PBS school is related to intended outcomes over time. (For 
example, over time, does a PBS school have an improved school climate, or reach a threshold 
level of positive school climate?) 

(3) Analyses using the outcomes that are likely to be most directly affected by PBS practices might 
be more sensitive. Past research (Bear, 2010) has suggested that suspension data is highly 
related to office referrals. In addition, the case studies suggested that there is some 
manipulation of the suspension and office referral numbers by in-school practices discouraging 
referrals. However, there may be other outcomes that are appropriate indicators of PBS effects. 

While these recommendations respond to some of the major flaws of our analyses, they do not address 
the more fundamental issue of the definition of "PBS" and what it means to be a PBS school in 
Delaware. 


Implications for Future Evaluations 

The findings from both the qualitative and quantitative components of the study point to next steps for 
evaluating DE-PBS. While the longitudinal analysis does not show significant suspension rate differences 
between PBS and non-PBS schools, the case study data suggest that this direct PBS/non-PBS comparison 
may not be the most appropriate. We know that PBS schools either volunteer or are volunteered to take 
on the PBS label, and that schools have varying levels of implementation success. Thus, variation occurs 
in how schools implement the core principles of PBS. Conversely, it is reasonable to assume that some 
schools that have not officially adopted PBS are implementing practices consistent with PBS approaches. 
If this is true, a more appropriate comparison would be between schools that implement practices 
consistent with PBS and those that do not. This suggests two follow-up studies. 

The first would be a measurement study to develop a rubric (or adapt existing measures of PBS 
implementation fidelity) that rates schools on how well student management in the school aligns with 
PBS principles. Whether this rubric rates schools on a continuous level or places them into categories, it 
could be applied equally to schools labeled PBS and those not labeled PBS, but possibly using PBS-like 
techniques. The second study would then build on the first with a quantitative analysis of effects on 
suspension rates (or other appropriate outcome measure) using the degree of fidelity with PBS 
principles - rather than whether the school was labeled as PBS or non-PBS - as the independent 
variable. In addition, the model could be used to explore whether additional school characteristics 
explain any differences in outcomes. 
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Postscript; Reflections and Future Directions 

George G. Bear 

As a consulting member of the PBS Core Team who has evaluated components of PBS in the past, I 
would like to make a few brief comments about the above findings. 

The results will certainly help guide the development of PBS programs in Delaware. However, most were 
anticipated. As found in this study, in previous years we also found no differences in suspension rates 
between PBS and non-PBS schools (Bear, Blank, & Pell, 2009). As noted earlier in this report, although 
one might hope to find fewer suspensions among PBS schools, there are several reasons (e.g., lack of 
random assignment and control for confounding variables), other than the possible ineffectiveness of 
PBS per se, why no differences were found and why reduced suspension rates might not be the best 
measure of program effectiveness. In the future, it might be more productive to examine changes over 
time within PBS schools on measures of positive student behavior and school climate. As recommended 
in this report, this might best be done by examining such changes as a function of the fidelity of 
implementation of the PBS key features and corresponding strategies and techniques. Although too few 
schools were included in the case study component of this evaluation to draw any conclusions, 
respondents' ratings on the Delaware Assessment of Strengths and Needs for Positive Behavioral 
Supports indicate that schools that implement the key features with the greatest fidelity are the ones 
with the fewest suspensions. Additionally, in my opinion, the most valuable findings come from the case 
study interviews. Consistent with research on school reform and school discipline (Bear, 2010), 
interviews indicate the importance of several factors: 

• the ongoing use of data for decision-making, 

• collaboration and communication, 

• shared commitment to program components, 

• consistency in the use of techniques across teachers and settings, 

• adapting and changing program elements over time (i.e., avoiding stagnation by viewing PBS 
not simply as "a tool to be pulled out of a bag" but as a complex, multi-component program), 
and 

• the active involvement of students in decision making. 


Whereas all of the findings are certainly important, I think two should be of primary focus for future 
improvements. First, whereas teachers, staff, and administrators often distribute tokens and rewards in 
PBS schools, many of them have significant concerns about their use. Respondents expressed 
philosophical and practical problems about the systematic use of tokens, including concerns about their 
age-appropriateness, consistency in their use across teachers and staff, their costs, their effectiveness in 
improving behavior, and the purposes of their use (e.g., using them to reward behavior that students 
readily exhibit without being given tokens). The second primary concern expressed by teachers and staff 
was that they receive little, if any, staff development and training in PBS. I suspect that this also is true 
with respect to school administrators (note that in the annual day-long workshop I have given on 
developing self-discipline - a key feature of DE-PBS -I have rarely found an administrator in attendance, 
and know that not all coaches have attended). Given insufficient staff development and training 
opportunities, and lack of attendance by key stakeholders, it is not surprising that PBS is often 
interpreted as being little more than the posting of expectations and use of tokens. 
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For years, members of the PBS Core Team have debated the use of tokens and rewards to manage 
student behavior and how to appropriately use them. They also have been well aware of the difficulties 
in providing adequate staff development and training opportunities, especially given the project's 
limited resources, the limited number of days and amount of time available for training, and the large 
number of schools that are now implementing PBS in Delaware. Although it is likely that the above 
concerns might have been predicted, it is unlikely that one would have predicted the frequency and 
vigor to which they were expressed. Clearly, these concerns require attention. 

Perhaps most troubling, however, is the finding that many teachers, staff, and administrators perceive 
PBS as synonymous with a token economy used to manipulate student behavior. This is not the message 
that the PBS project has intended to communicate to educators, especially in recent years. It should be 
noted that the current evaluation did not include DE-PBS schools that received the most recent training 
provided to new DE-PBS schools. Nevertheless, how and why this lack of communication, or 
miscommunication, has occurred, and how to correct it, needs to be addressed by the project. It is 
equally troubling that many educators fail to recognize that the primary goals of DE-PBS are to develop 
student self-discipline and foster a positive school climate, including positive teacher-student, student- 
student, and school-home relations. It appears that in many schools the project's goals and key features 
have not been communicated or translated effectively into more comprehensive practices. It is likely 
that the project will find it difficult to change the (faulty) perception among many teachers and 
administrators (and perhaps some PBS coaches) that DE-PBS is little more that the distribution of 
tokens. A failure to change this perception, however, may very well undermine implementation of 
Delaware's new key features. Thus, perhaps the greatest challenge to DE-PBS with respect to 
establishing programs of demonstrated effectiveness will be to persuade schools that there is much 
more to DE-PBS than the use of tokens to manage student behavior. The project's recent efforts in 
creating measures and staff development opportunities linked directly to its new key features should 
help meet this challenge. 

Given that a limited vision of PBS exists in many schools, change will not be easy, especially with sparse 
resources and few opportunities for staff development and training. Nevertheless, the following 
recommendations are proposed (in addition to the eight recommendations made on pages 13 and 16) 
to help change the perception of PBS as simply the use of tokens to an understanding that the primary 
aims of PBS in Delaware are to develop self-discipline and a positive school climate (with or without the 
use of tokens). 

(1) More extensive and on-going training should be provided, especially that which focuses on 
recent changes in DE-PBS philosophy and recommended practices (i.e., the key features and the 
practices recommended in the Delaware Assessment of Strengths and Needs for Positive 
Behavioral Supports). This would include best practices related to each of the factors identified 
by the interviews as being important to the effective implementation of PBS (see page 7). 

(2) Whether or not staff development and training for teachers, staff, and administrators is best 
provided via the current "train the coaches" model should be closely examined. Changes in this 
model, as needed, should be made. Problems with this model are (a) not all coaches are 
required to attend all of the PBS training sessions, (b) there is little effort to monitor the extent 
to which the information and skills presented in statewide training sessions are actually 
communicated/taught to teachers, staff, and administrators in PBS schools, and (c) there is a 
high risk of resistance among teachers, staff, and administrators, especially if they are not active 
participants in the decision-making process (e.g., it is expected that coaches will train teachers, 
staff, and administrators in skills that they are assumingly lacking - an assumption that is likely 
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to lead to resistance, especially when the recommended techniques are not consistent with 
one's existing philosophy and practices). If the current training model continues, increased 
efforts should be taken to help ensure (a) consistency in the information/skills delivered to the 
PBS coaches in their trainings and the information/skills delivered /rom the coaches to teachers, 
staff, and administrators; (b) consistency in training with the DE-PBS key features and evaluation 
measures, and (c) consistency across training workshops and sessions, and (d) the active 
involvement of teachers, staff, and administrators in all decisions pertaining to DE-PBS and 
adapting it to meet the philosophy and needs of each individual school. 

(3) In helping persuade schools that PBS is not just about behavioral expectations and tokens, the 
project should place increased emphasis on achieving valued student and school outcomes, 
especially developing self-discipline and promoting a positive school climate. Multiple strategies 
and techniques for achieving those outcomes should receive primary attention, and they would 
include the wise and strategic use of rewards (e.g., not using them in ways that research shows 
might undermine self-discipline and a positive school climate). 

(4) Building-level administrators should be required to attend training in PBS, such as a two-day 
workshop that focuses on the key features and the strategies and techniques highlighted in the 
Delaware Assessment of Strengths and Needs for Positive Behavioral Supports. It is likely that 
leadership in DE-PBS is lacking in schools in which administrators have little understanding, or 
unclear understanding, of the project's goals and the best practices for achieving them. 

(5) In light of serious barriers to providing increased staff development, training and technical 
assistance, including limited resources and training opportunities, the PBS project should 
consider targeting a small number of schools. It might be politically unwise, however, to remove 
schools from the project. One way to avoid this problem and provide more focused training and 
technical assistance to a smaller number of schools might be to allow many current PBS schools 
to continue with their current practices (without the same level of training and assistance). 

(6) Increased efforts should be made to ensure that staff development workshops and other 
trainings and technical assistance provided to PBS schools, are directly aligned and consistent 
with the DE Key Features as well as the strategies and techniques highlighted in the Delaware 
Assessment of Strengths and Needs for Positive Behavioral Supports. Likewise, trainings and 
technical assistance should be provided that are more tailored to the specific needs of individual 
schools, such as needs revealed in the strengths and needs assessment, and scores on the 
Delaware School Climate Surveys and other important outcome measures (e.g., suspension 
rates). This might include training targeted specifically for improving areas of school climate 
and/or developing self-discipline. Comprehensive strategies and techniques for achieving those 
two aims would be the focus of on-going staff development, reflection, and training. 

It should be noted that each of the above recommendations are incorporated into the $4 million "Safe 
and Supportive Schools" grant proposal that members of the PBS team are now writing. 
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Appendix A: Detailed Description of Methods for Question 1-Contributors and Barriers 


Interview data collection and analysis 

Administrator, PBS team, teacher, student, and district coach interviews 

With the exception of interviews with the district PBS coaches, which were scheduled directly, school 
interviews were scheduled through the school principal or another designated administrator (eg. vice 
principal, PBS team leader). In all schools, the coordinating individual chose to distribute the consent 
and assent forms to all potential participants and to arrange dates, times, and locations for them. 
Consent forms were collected from all interviewees except administrators. As public officials, they could 
not be offered the same assurances of confidentiality as the other participants. 

In April 2010, researchers from DERDC conducted interviews with 130 school administrators (typically a 
principal or assistant principal), PBS district coaches, members of PBS school teams, and teachers not on 
PBS teams (See Table 6 for details on participants). Administrators and coaches were typically 
interviewed individually; PBS team members and teachers were usually interviewed in small groups. In 
addition, a group of students was interviewed at each of the four schools with a low incidence of 
disciplinary infractions. The interviews were semi-structured with questions focused on the nature of 
PBS in each school and on interviewees' perceptions of contributors and barriers to the implementation 
of PBS. The interviews with staff lasted approximately one hour, and the student interviews were about 
30 minutes long. With the exception of one of the student focus groups, all interviews were audio 
recorded and transcribed. 

Participants in the PBS team and teacher focus groups, as well as school administrators, were asked to 
complete the Delaware Assessment of Strengths and Needs for Comprehensive School Discipline. A gift 
debit card of $75 was provided as an incentive to PBS team members and teachers to participate in the 
focus group interview and fill out the survey. 

Table 6. The number of interview participants at each school, A-F 



School 

Respondent Type 

A 

B 

C 

D 

E 

F 

Administrators 

1 

2 

2 

1 

1 

2 

Teachers 

9 

9 

8 

9 

1 

5 

PBS Team 

9 

5 

7 

7 

8 

3 

Students 

- 

- 

11 

11 

4 

10 

PBS Coach 

1* 

1* 

1 

1 

Total 

20 

16 

29 

29 

15 

21 


*Schools with the same PBS coach. 


An adaptation of the Center for Disabilities Studies list of key elements of PBS in Delaware served as an 
initial coding framework for the interview transcripts. We also developed codes to label instances when 
participants spoke of contributors and barriers that were not represented in the list of key factors. For 
each school, the coded text was compared across respondent groups. In addition, we looked at each 
respondent group (e.g., administrator, teacher) across all schools. Through this comparative analysis we 
identified the most prominent contributors and barriers to PBS implementation as described across all 
schools and groups of interviewees. These are detailed in the main body of the report. 
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Appendix B: Questions Used in Administrator, Coach, PBS Team, Teacher, and Student Interviews 


The interview questions used to collect data in the schools selected for the case studies were semi- 
structured and intentionally open-ended. Thus these questions served as guides and were not followed 
verbatim or in order. The interviews were also constrained by time, so not all questions were covered in 
all the interviews. Those questions we gave the highest priority are indicated by bold font. 

Coaches Interview Questions 

1) How long have you been a coach for this school? 

2) What are the major components of PBS in this school? 

Probes if not mentioned: 

a) What, if anything, is the school doing in terms of proactive and preventive components? 

b) What, if anything, is the school doing in terms of corrective components? 

c) What, if any, supports are provided at each of the 3 tiers (SW, targeted/group, and individual)? 

3) To what extent, if any, would you say the school is engaged in data-based decision making? Please 
give me an example. 

4) Qverall, how would you assess the school's implementation of PBS? Would you say that it had 
fully implemented PBS, partially implemented PBS, or minimally implemented PBS? [If partially or 
minimally, ask for more information.] 

5) You completed a survey about the technical assistance provided to this school and indicated that TA 
in the first two years was [rating] and TA in years 3 onward was [rating]. I'd like to know more about 
the specifics of the TA you provided. Please describe what kinds of support this school received in 
the early stages of implementation. [Interviewer to check these off as mentioned, probe on any not 
mentioned, and add any mentioned that aren't on the list. Interviewer will also probe for more 
detail/examples.] 

□ Made presentations to all staff and grade level or department meetings 

□ Met with administrator to plan process 

□ Active in new team recruitment and team leader selection 

□ Provided SW PD to the team 

□ Attended and worked with team during state SW PD workshop 

□ Supported school in accessing data reports or provided data reports to the school 

□ Supported school in analyzing data 

□ Provided assistance with products (review, creation, etc.) 

□ Attended monthly school PBS meetings 

□ Prepared school for SET-D - understanding the purpose 

□ Reviewed results of SET-D and recommendations with the team 

□ Other(s): 

6) Please describe the kind of support that the school has received, when it was maintaining PBS 
implementation in years 3 and after. [Interviewer to check these off as mentioned, probe on any not 
mentioned, and add any mentioned that aren't on the list. Interviewer will also probe for more 
detail/examples.] 

□ Check-in with team leaders 
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□ Meet with administrator 

□ Attend monthly school PBS meetings 

□ Provide assistance with products review and/or revisions 

□ Support schools in accessing and analyzing data 

□ Facilitate additional team trainings for PBS team members 

□ Support team with whole faculty presentations or PD on PBS related topics 

□ Other(s): 

7) Looking at both TA support and more broadly, what would you say has facilitated PBS 
implementation at [school]? 

a) PBS-related facilitators? 

b) Other (non-PBS) facilitators? 

8) What obstacles have been encountered in implementing PBS? Were those obstacles resolved? If 
so, how? 

a) PBS-related obstacles? 

b) Other (non-PBS) obstacles? 

9) Some things mentioned as facilitators or challenges are listed on this card. To what extent would 
you say each of these items acted as a facilitator or obstacle or neither? [Also ask respondent to 
rate anything facilitators/obstacles that s/he mentioned in response to Qs 7 and 8 that don't 
appear on the card.] 


Potential 

facilitator/obstacle 

A major 
obstacle 

Somewhat of 
an obstacle 

Not a factor 
either way 

Somewhat of a 
facilitator 

A major 
facilitator 

Staff commitment and 
participation 






Leadership of team (team 
leader and/or admin) 






District support for program 






TA provision 






Staff development 







10) If time allows: If the school had the resources to address one of the obstacles, which do you think 
they should address first? 

11) In your opinion, has PBS had an impact in this school? [If yes] Please describe its effects. 

12) What outcomes would you have expected to see in this school by now, but haven't? Why do you 
think those haven't occurred? 

13) If time allows: Here is a list of intended impacts. On a one to five scale, please tell me how much of 
an impact, if any, you think PBS has had on each one of these. 



Little or 
none 

Some 

Moderate 

Great 

Very 

great 

Intended impact: 

1 

2 

3 

4 

5 

Preventing behavior problems 






Reducing office disciplinary referrals and school 
suspensions 
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Developing self-discipline or social and emotional 
competencies 






Increasing the use of data in decision-making 






Building staff morale 






Improving teacher-student relationships 






Increasing school safety 






Improving overall school climate 







Administrators Interview Questions 

1) What is your role with PBS in the school? 

a) How long have you been involved? 

b) What professional development did you receive in PBS? 

c) Were you involved with PBS in other schools? 

2) What are the major components of PBS in this school? 

Probes if not mentioned: 

a) What, if anything, is the school doing in terms of proactive and preventive components? 

b) What, if anything, is the school doing in terms of corrective components? 

c) What, if any, supports are provided at each of the 3 tiers (SW, targeted/group, and individual)? 

3) To what extent, if any, would you say the school is engaged in data-based decision making? Please 
give me an example. 

4) Overall, how would you assess the school's implementation of PBS? Would you say that it had 
fully implemented PBS, partially implemented PBS, or minimally implemented PBS? [If partially or 
minimally, ask for more information.] 


5) Some things mentioned as facilitators or challenges are listed on this card. To what extent would 
you say each of these items acted as a facilitator or obstacle or neither? 


Potential 

facilitator/obstacle 

A major 
obstacle 

Somewhat of an 
obstacle 

Not a factor 
either way 

Somewhat of a 
facilitator 

A major 
facilitator 

Staff commitment and 
participation 






District support for 
program 






TA provision 






Staff development 






Team leadership 






Program evaluation 







6) Has this school's PBS program experienced any facilitators or obstacles not on this card? What are 
they? 

a) PBS-related obstacles? 

b) Other, non-PBS obstacles? 
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7) If time allows: If you had the resources to address one of the obstacles, which would you work on 
first? What other ways would you improve implementation? 

8) What effects - positive or negative - do you think PBS has had in your school? 

9) If time allows: Here is a list of intended impacts. On a one to five scale, please tell me how much of 
an impact, if any, you think PBS has had on each one of these. 



Little or 
none 

Some 

Moderate 

Great 

Very 

great 

Intended impact: 

1 

2 

3 

4 

5 

Preventing behavior problems 






Reducing office disciplinary referrals and school 
suspensions 






Developing self-discipline or social and emotional 
competencies 






Increasing the use of data in decision-making 






Building staff morale 






Improving teacher-student relationships 






Increasing school safety 






Improving overall school climate 







10) What outcomes had you hoped to see by now, but haven't? What would you attribute that to? 

11) If time allows: If you could give other schools one piece of advice about implementing PBS, what 
would it be? What about the District office? The state? 

12) Is there anything else that you think I should know about the implementation or impact of PBS in 
your school? 


PBS Team Focus Group Questions 

1) What are the major components of PBS in your school? 

a. What specific practices (if any) have changed as a result of PBS? 

b. In what ways (if any) are data used in decision-making in the school? 

c. What has been the role of the PBS team in the school? 

2) What obstacles to implementing PBS have you encountered? In what ways (if any) have these 
obstacles been addressed? 

3) What (if anything) has facilitated implementation? 

4) What would facilitate or strengthen the implementation of the program now? Probes, if time 
permits: 

a. Staff commitment and participation 

b. Leadership of PBS team 

c. District support for program 

d. Technical assistance provision 

e. Staff development 
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f. Program evaluation 

5) If you had the resources to address one of these areas, which would you focus on first? 

6) What impacts, if any, has PBS had on the school? 

7) What outcomes did you hope to see by now, but haven't? What would you attribute that to? 

8) If time allows: What recommendations or advice would you give to other PBS schools, the district 
office, or the state about how to implement PBS? 

9) Is there anything else that you think I should know about the implementation or impact of PBS in 
your school? 

Teacher Focus Group Questions 

1) Please tell me your name and what position you have in the school (e.g., grade, subject, specialist, 

...). 

2) What are the major components of PBS in your school? 

3) What specific practices, if any, have changed as a result of PBS? 

4) In what ways, if any, are data used in decision-making? 

5) What effects - positive or negative - do you think PBS has had in the school? 

a) How, if at all, has PBS affected your teaching practices and/or classroom management? 

b) How, if at all, has PBS affected your relationships with students? 

6) What has facilitated PBS implementation? 

a) PBS-related? 

b) Not PBS-related? 

7) What obstacles have been encountered in implementation? 

a) PBS-related? 

b) Not PBS-related? 

8) What professional development (if any) have you received in PBS? 

a) Were there any that were particularly useful? In what ways were they useful? 

b) Were there any that were not that useful? What kept them from being useful? 

9) What other kinds of supports have been provided for PBS in the school? 

10) If there were resources to address one obstacle or strengthen an element of PBS, which would you 
choose to focus on first? 

11) Is there anything else that you think I should know about the implementation or impact of PBS in 
your school? 

Student Focus Group Questions 

1) What does the PBS program do in your school? Make sure get a full answer to this guestion, 
following up with "...anything else?", "can you give me an example?" "I'm not sure what you mean 
by that, could you tell me a little more?" 

2) Do you think it's working? In what ways? 

3) What do you like the most about PBS? 
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4) What do you like the least about it? 

5) If you were the principal, what would you do to improve PBS? 

6) What else would you do to improve the school for students? 

7) What do you think is the most important thing teachers do to help keep students from 
misbehaving? 

8) What do teachers do when there are problems in class, like talking back or making trouble? 

9) For middle school students: Have you ever been asked by your teachers, principal, or in a survey 
about [the program], like how it should work, how to improve it, or if you like it? Can you tell me 
more about that? 

10) What are other things about PBS that you think I should know? 
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Appendix C: Delaware Assessment of Strengths and Needs for Positive Behavior Supports 


After the interviews, participants in the PBS team and the teacher focus group interviews were asked to 
complete an online version of the "Delaware Assessment of Strengths and Needs for Positive Behavior 
Supports" (Bear, Burwell, Baker, Blank & Boyer, 2009). With items aligned with the key features of DE- 
PBS and drawn from the research literature on school discipline and classroom management, the 
instrument is designed to help schools assess and examine their strengths and needs related to 
schoolwide and classroom discipline efforts. Using 5-point scales, respondents are asked to rate the 
importance of items and indicate the extent to which each item is a strength or weakness at the school 
and classroom levels. 

At two schools with high incidence of disciplinary infractions, 31 of 37 (84%) staff members completed 
the survey. Thirty of the respondents indicated their roles in the school: 80% are classroom teachers, 
3% are administrators, and 17% are other personnel. The results are presented in Figure 2. 



Promoting Developing Correcting 
Positive Self-Discipline Misbehavior 
Behavior 


Addressing 
Needs of 
Students 


Program 
Development 
& Evaluation 


■ Importance 

■ Schoolwide 

■ Classroom 


Assessment Components 


Figure 2. Survey results from two schools with a high incidence of disciplinary infractions (n=31 respondents) 


At four schools with low incidence of disciplinary infractions, 49 of 55 (89%) staff members completed 
the survey. Forty-eight of the respondents indicated their roles in the school: 65% are classroom 
teachers, 6% are administrators, and 29% are other personnel. The results are presented in Figure 3. 
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Figure 3. Survey results from four schools with a low incidence of disciplinary infractions (n=49 respondents) 
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Appendix D: PBS Technical Assistance Survey 


This survey asks you to estimate the amount of technical assistance in PBS you have provided to specific 
schools. Please contact Cheryl Ackerman at 302-831-8585 or cma@udel.edu if you have any questions. 
Thank you in advance for your time and assistance. 

Ql. Technical Assistance during initial PBS Implementation 

The first question asks about technical assistance provided in the first two years of PBS participation, 
when the school is preparing for implementation. During this time, typical TA activities include: making 
presentations to staff, conducting planning meetings, helping recruit PBS team members and select the 
team leader, providing school-wide support to the team, working with the team during the state school- 
wide professional development workshop, supporting the school in accessing and analyzing data, 
assisting with the creation and review of products, attending monthly PBS team meetings, preparing the 
school for the SET-D, and reviewing the results of the SET-D with the team. 

For each school listed below, indicate how much technical assistance support was provided during their 
beginning implementation (during the first two years in the program), with "1" equal to minimal support 
and "7" equal to an extensive level of support. 



Little or 
None 
1 

2 

3 

Average 

4 

5 

6 

Extensive 

7 

School 1 (200X-2009) 








School 2 (200X-2009) 

















Q2. Technical Assistance During PBS Maintenance 

This question asks about TA provided during the maintenance phase of PBS implementation (years 3 and 
after). During this time, TA activities can include: checking in with team leaders, meeting with the 
administrator, attending monthly school PBS meetings, assisting with product review and/or revision, 
supporting school in accessing and analyzing data, facilitating additional team trainings, and supporting 
team with whole faculty presentations on PD or PBS-related topics. 

For each school listed below, provide a global rating of how much technical assistance was provided in 
years 3 and after, with "1" equal to little or no TA and "9" equal to an extensive TA. The specific years 
for each school are included below. 



Little or 
None 
1 

2 

3 

Average 

4 

5 

6 

Extensive 

7 

School 1 (200X-2009) 








School 2 (200X-2009) 
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Q3. Additional comments 

Please provide any additional information you think we need to understand your ratings. For example, if 
there were special circumstances at a school that resulted in the school needing significantly less or 
significantly more TA than average, please briefly describe those circumstances. 

School 1: 

School 2: 
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Appendix E: Illustrative Quotes for Contributors and Barriers Themes and Sub-Themes 


The following tables provide quotes illustrating the themes and subthemes discussed in the findings for 
the first evaluation question - contributors and barriers to implementing PBS. Each quote is followed by 
a letter (eg. (A)) which indicates whether the statement was made by an administrator (A), coach (C), 
teacher (T), PBS team member (TM), or student (S). 


Table 7. Contributors and barriers in the consistency and adaptability theme 


Contributors 

Illustrative Quotes 

Consistent commitment to PBS 
by teachers and administrators 

- "They have to have buy-in - that's important so you have to get people to 
understand why it's important and why you're doing it " [A] 

- "1 think one of the things that has really facilitated the success of it is the buy 
in from all the staff, you know from the Principals to the teachers to the 
cafeteria workers. Just everyone has bought into it and 1 think that's what 
makes it successful" [TM] 

- "1 think they need a committed core team of people who will come back 
every year and refresh that commitment to implement positive behavior 
support." [C] 

Consistency of policies and 
practices as applied in the 
school 

- "Those rules are throughout the whole school, not only in the classroom, but 
it's being safe in the hallway, in the playground, on the bus, and in the 
cafeteria. ..The rules are the same no matter where you are and no matter 
what grade you are in." [T] 

- "In order to keep it consistent, the schoolwide is necessary so that you don't 
have kids saying 1 have 10 [rewards] and 1 don't have the other 5 because so 
and so never gives them out. ..You know a good behavior in this classroom 
should be the same good behavior in another one." [TM] 

- "Some teachers try to tweak it or experiment with it one way or another 
then that deteriorates the consistency throughout the building." [TM] 

Willingness to adapt and 
experiment within the 
framework of PBS 

- "So 1 think one thing we're always doing is trying to say like 'where are we?' 
and 'what can we do to improve it?"' [A] 

- "When things are getting stale we're willing to look at things so that we're 
keeping it fresh " [TM] 

- "This school is not afraid to try things and sometimes we try things and they 
don't work and then we fix it or we scratch it and go back to the way we 
were before." [TM] 

- "1 feel like we look at the data and we're always changing, we're always 
tweaking something one way and tweaking something another way and to me 
that is that we're constantly evaluating the program. It's on-going evaluation." 
[A] 

Barriers 

Illustrative Quotes 

Lack of buy-in to PBS 

- "There are still considerable people, teachers and staff, who are resistant and 
that clearly affects how well they are able to implement. There are some 
naysayers that clearly don't want to be on board." [C] 

- "They have to have buy-in - that's important so you have to get people to 
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understand why it's important and why you're doing it so communicating 
why this is important to the school - why we're doing it and getting people 
to commit to it." [A] 

- "But as far as using it to enhance classroom management, not at all. We 
support the activities that are out there but it's not used for day in and day 
out classroom management. ..and 1 wouldn't use it personally." [T] 

Limited understanding of PBS 
principles 

- "...something we're constantly working on is people understanding it [the 
currency used as rewards] has to be one at a time. .That's an education that 
goes on every month it seems like. Being sure that people use them the way 
they're supposed to be used." [A] 

- "...our problem with our staff is folks that aren't trained in PBS and. ..they get 
panicky if they feel they are losing control of a behavior and they want to use 
it as a hammer as opposed to facilitating." [A] 

- "There are some folks out there that are still not exactly sure of the basic 
philosophy. " [TM] 


Table 8. Contributors and barriers in the rewards theme 


Contributors 

Illustrative Quotes 

Student input 

- "With the school store it has changed because of the survey that was taken 
by fifth grade students and they came up with different things they would 
like to see in the school store." [T] 

- "We decided we wanted to make sure that prizes were something kids you 
know really wanted to earn." [A] 

- "They don't ask us what kind of parties we have. ..they just make us do 
whatever they think we should have. Like we should do like. ..a. ..they should 
give us options. ..because some things that we do. ..aren't that fun 
sometimes." [S] 

Appropriate nature and timing 
of rewards 

- "Another thing that is always told to us is to keep the rewards simple 
because you don't want the students thinking that if I'm good 1 can get a 
reward. You want them to think 1 should do the right thing. So we always try 
to keep the rewards really simple." [TM] 

- "1 thought a brilliant, really great idea was to do the dress up dress down day 
because these kids all they want to do is 1 mean across the school is to dress 
down. ...and it was one of those things that didn't cost us anything but that 
was the best ...it was worth it...." [T] 

- "Certain things occur like we have dances or sports day - different events 
held once every two months." [TM] 

External support 

- "We have a formal letter that we use and we provide that to each 
community partner, non-profit or profit. It explains our program and we 
basically get the funding and resources just by our letter.... Another example 
of a donation or a pitch would be to go to the local WaWa and students that 
have the best behaved classroom, and there's no discipline referrals in that 
homeroom for that month, they would receive a free milkshake from 
WaWa. [A] 

- "We seek help from outside agencies and from businesses. Walmart 
donated a thousand dollars. ..The money doesn't just grow on trees and it 
doesn't come out of the school budget. A major thing on that committee is 
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fundraising and those teachers do that. ..They have a Friendly's night where 
people show up and we get a certain percentage of the proceeds and 
they're always doing stuff." [A] 

Barriers 

Illustrative Quotes 

Lack of clarity/agreement 
about when to use rewards 

- "...1 feel as though not every teacher gives the students the tickets for the 
same behaviors. It's not consistent among teachers." [TM] 

- "The obstacles that we've come across with our economy. ..something we're 
constantly working on is people understanding it has to be one at a time, it 
can't be at the end of the week give somebody 20. It needs to be a constant 
feed." [A] 

Student manipulation of the 
system 

- "1 just feel that as the kids are going from grade to grade they're learning 
how to manipulate the system so by fifth grade 1 mean I've had a kid come 
up to me and say put his feet on my desk and say let's make a deal." [T] 

- "They can manipulate the system...! just think there needs to be consistency 
[but] 1 don't know how to develop it. It's not flawless and 1 just think that 
there needs to be more of a way if they're switching from room to room." [T] 

Rewards that are inappropriate 
for the age group 

- "One obstacle is getting the 8**^ graders to stay on board. Some of them think 
they're too cool for it. They don't want to participate. You give them the 
[currency] and they throw it in the trash or give it to their friend." [TM] 

- "...sometimes they [T**^ and 8**^ graders] feel as though it's a 6**' grade event 
and it's not cool to sign up or go to that because my buddies aren't 

going. ..because there's a lot of kids that will receive enough tickets, it's just a 
matter of them wanting to participate. In 6*'^ grade we could go play in a mud 
puddle, they'll be there. 8**' grade, they are selective." [T] 


Table 9. Contributors and barriers in the data-based decision-making theme 


Contributors 

Illustrative Quotes 

Responding to emerging issues 

- "... certain times of the year we know there is going to be an increase in 
referrals because of whatever outside factors, 1 think that is the whole point of 
looking at data so you can then plan the most appealing or most alluring 
events around that time to try to combat the inevitable." [TM] 

- "We do a lot with data. Our PBS team meets monthly and then two half day 
sessions and we always look at our discipline data and talk with the teachers 
about what they're seeing in the classrooms and around the school. Then we 
determine where our needs are." [A] 

- "When we look at our discipline data, you can see what it is, if there is a 
particular grade level, if there is a particular child - we definitely you know if 
there's a location. ..breakfast used to be a disaster so we added another staff- 
pulled a staff person from one area where there's no referrals and put them in 
the cafeteria where there are referrals but that's basically how we use data." 
[TM] 

Monitoring individual behavior 

- "We look at our monthly offenses (level 2, 3, 4). We look at where they are 
occurring, are there repeat offenders, and what have we done for those 
repeat offenders. Are they seeing the guidance counselor, so they have an 
FBA (Functional Behavior Assessment), do they have an lEP that has 
accommodations. Are we meeting those?" [A] 

- "We use this data and this is just a wealth of data. 1 mean 1 can tell you who is 
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in detention today and 1 can say to the detention proctor, give a ticket 
[reward] to your best three kids." [A] 

Increased staff buy-in 

- "But some of our teachers weren't buying into it as much as they could have 
for a lot of reasons. One is because they didn't see the benefits of it. Well, 
now they see the benefits of it because we are able to track it and get a little 
bit better, and we are able to use it a little bit better, so now we have more 
teachers buying into it." [A] 

- "We look at it as a team and then [the assistant principal] shares it with the 
school improvement team who has a rep from each grade level and through 
that grade level rep it's fed back to all the staff." [TM] 

Barriers 

Illustrative Quotes 

Data as window-dressing 

"...we have the academic data posted all around in our classroom, down stairs, 
when you walk in the buildings, you hear about it during staff development 
meetings, but we never hear about the school climate data. And the referrals, 
that data is not posted." [T] 

Unsystematic collection of data 

"1 would ask them - where's your data? First, it was not even there to begin 
with. Then they started bringing data, but 1 think that they thought all they had 
to do was look at it instead of analyze it. Now 1 think they are getting it; it's 
emerging. 1 think they're improving, but that has always been a major roadblock 
in their ability to reflect on what they're doing and make decisions based on the 
data about what needs to be done differently. " [C] 


Table 10. Contributors and barriers in the professional development and support theme 


Contributors 

Illustrative Quotes 

District Coach as a resource 

- "At this point we felt we haven't needed our district coach, she's there if we 
need her, but things have been running smoothly. " [TM] 

- "1 was very, very involved with them when they first started. 1 haven't been 
nearly as involved the last few years because it starts to be institutionalized 
and they don't need the same level of support." [C] 

- "Is there still a District Coordinator?" [TM] 

Annual refresher training 

- "We actually revamped our program. ..and so we went to a retreat where we 
spent a good part of the day retraining all the new staff so everybody had a 
refresher." [A] 

- "If there are any new staff they just need to keep it fresh and keep it you 
know at the forefront of what they are doing." [C] 

- 1 think it is necessary every year to cover the whole thing - the basics, the 
philosophy, the way it works. ..There is some time that should be invested in 
making it work every year and we just haven't been able to put the time into 
because of all the other constraints in the building. [T] 

Barriers 

- Illustrative Quotes 
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Insufficient training 

- "I'm a first year teacher here and I've never gotten any training. 1 spent the 
first three weeks, they gave me tickets [reward currency], and 1 had no idea 
what they were." [T] 

- "1 feel like if we had more professional development we wouldn't have so 
many inconsistencies." [T] 

- "There was training at one point and some of us were told that we were 
going to go and then we were told that we couldn't go and 1 don't know what 
the reasoning was and who actually went but 1 know 1 have never had any 
training on it." [TM] 

- "1 think at [school name] there's a degree of complacency that's kind of set 
in with the program because it has been so polished for so long 1 think it 
needs to be somehow refreshed" [C] 

Limited resources 

- "That is how the District could help us - ...provide substitute teachers so that 
you know that everyone can get the state wide training that PBS offers. " 

[TM] 

- "...we don't have a whole lot of staff development days. 1 would guess that in 
past years when they've offered a selection on professional development 
days PBS has been incorporated into that. ..but we don't have that. PBS does 
come up at faculty meetings and through conversations with teachers but it's 
not specific to staff development." [A] 


Table 11. Quotations illustrating the students' perspectives. 


Topics 

Illustrative Quotes 

Rewards 

- "The rewards are more for like middle school interests. Like in middle school 
they're not going to want to be winning candy they're going to want to win 
money and stuff that they can actually use." 

- "If 1 was Principal 1 would give out more [school currency] because if you give 
out [currency] it will encourage the kids to do better." 

Consistency 

- "1 would say that most teachers don't really realize that some kids are doing 
something good. 1 would get teachers to notice more and to point kids out to 
say they're doing a good job and stuff." 

- "Some of the teachers are strict, they use PBS against us. Like they'll take 
away tickets. Some teachers don't use tickets, or they'll give them if you pass 
a test. One teacher will give them if you sharpen your pencil [said with a 
mocking tone]! The bus driver can give you tickets for behaving on the bus." 

- "Say if you stay after class and wash the board some will give you ... bucks. 
Other teachers might not do that but they might give you something for 
struggling with a problem and then you finally get it right. They might give you 
... bucks to say congratulations for studying." 

Corrective Measures 

- "More discipline, because the same people that get in trouble get in trouble 
again because they don't have harder discipline." 

- "And then they give you slips to give to your parents telling what you did and 
they make sure that your parents sign it. And then you get disciplined if you 
don't. Like you get detention or something and they make sure that your 
parents sign it and then you give it back to the teacher that gave you 
detention. So that's how they keep track of like little stuff." 
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Appendix F: Details of Analyses for Question 2 - Disciplinary Infractions 


This appendix describes the models used in the analysis of differences in suspension rates between PBS 
and non-PBS schools. In addition, analyses were planned with data from the Delaware School Climate 
Survey. School climate data for the 2008-2009 year were provided by the Center for Disabilities Studies' 
Positive Behavior Support initiative. We examined the possibility of comparing the school climate scores 
of PBS and non-PBS schools. However, longitudinal analysis was not possible since the climate survey 
has changed from year to year. 

Longitudinal Growth Model 

Although PBS is not randomly assigned to schools as a cluster randomized trial, we can still model the 
data looking for differences while keeping in mind the third variable problem. To account for the nested 
structure of the data (suspension rate data nested within schools over time), hierarchical linear 
modeling (HLM) is used. HLM is a generalized form of regression analysis that is used to account for and 
explicitly model the impact of nested structure and the resultant correlated residuals (Raudenbush & 
Bryk, 2002). This technique allows for unbiased estimation of standard errors as compared to using 
ordinary least squares methods with nested data. HLM software version 6.08 is used for this analysis 
(HLM for Windows, 2009). 

First, the unconditional model is specified to calculate the intra-class correlation of suspension rates 
within schools: 

Level 1: SUSRATEit= noi+ eitwith eit"" N(0, o^) 

Level 2: noi = (3oo+ roi 
Combined: SUSRATEit= (3oo+ foi+ e:t 

Next, we estimate a growth model for the repeated measures of suspension rates, allowing for random 
effects between both intercept and slope and allowing these components to covary. The first year, 

2005, is considered time=0 to make the intercept meaningful, and the time variable is represented as 
TIMEO. Our generalized growth model is: 

Level 1: SUSPENi,= noi + niiTIME0,+ eitWith eit~ N(0, o^) 

Level 2a: noi= Poo+ PoiWj+ roi 

Level 2b: nii= Pio+ PnWj + ri. 

Although the model is simpler to visualize as separated into level equations, the final model actually 
combines both levels and runs only a single model: 

Combined: SUSPEN|,= Poo+ PoiWj + PioTIMEOt+ PnWjTIMEOt + TIMEOtrii + roi+ eit 

For each variable of interest, we now test whether the main treatment effect Wj of PBS is statistically 
significant by testing whether: 

For intercepts, Hq: Poi= 0 or Hi: Poi^ 0 

For slopes, Hq: Pn= 0 or Hi: Pn^^ 0 

Model Assumptions 

Basic regression assumptions are checked for both elementary and middle school models. Examination 
of graphed data suggests a linear relationship between suspension rates and time. Plotting level 2 
residuals shows approximate normality and no worrisome outliers in either case. Checking for 
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homogeneity of level 1 variances in elementary (x^=1.36, df=51, p>0.50) and middle schools (x^=0.29, 
df=17, p>0.50) shows no significant variance differences across schools. 

We also explore whether PBS and non-PBS schools have different variances. We estimate a 
heterogeneous sigma squared predicted by PBS membership and compare the deviance from this model 
to the unconditional model. The deviance statistic is significant for elementary (x^=39.61, df=l, p<0.001) 
and middle schools (x^=11.94, df=l, p=0.001), suggesting an additional variance component should be 
estimated. Having a heterogeneous sigma squared changes the interpretation of the elementary model, 
yet not the middle school model. The elementary school model contrast indicator becomes significant 
(Pii=0.036, p=0.22), suggesting PBS schools have a change with higher suspension rates over time 
compared to the non-PBS schools. Due to the small and equal sample sizes, the simpler model assuming 
homogenous variance is run and those results are presented. 

Detailed Results of Longitudinal Growth Model 

Elementary Schools 

A total of 55 elementary schools fit the eligibility criteria, with 17 (31%) not implementing and 38 (69%) 
implementing PBS. The intraclass correlation coefficient, the variance in scores resulting from group 
membership, is 0.59. This statistic can be thought of as the average correlation between two time points 
in suspension rates, and shows how consistent suspension rates are within a school. The geographic 
characteristics of elementary schools included in this analysis are shown in Table 12. Results for the 
elementary school growth model are shown in Table 13. 

Table 12. Geographic characteristics of PBS and non-PBS elementary schools included in analysis 



PBS 

Non-PBS 

Counties 



Kent 

3 

5 

New Castle 

28 

11 

Sussex 

7 

1 

School Districts 



Appoquinimink 

5 

0 

Brandywine 

3 

1 

Caesar Rodney 

0 

1 

Cape Henlopen 

4 

0 

Capital 

2 

4 

Christina 

12 

0 

Colonial 

3 

2 

Lake Forest 

1 

0 

Laurel 

2 

0 

Red Clay 

5 

8 

Seaford 

0 

1 

Woodbridge 

1 

0 


The average suspension rate in 2005 for non-PBS schools is significantly different from zero. However, 
the PBS impact on this intercept is not significant, meaning that PBS and non-PBS schools do not have 
different starting points in suspension rates in 2005. The fixed effect for the slope is not different from 
zero, meaning that non-PBS schools are not showing any significant change in suspension rates over 
these four years. The impact of PBS on this slope is not significant either, meaning that PBS schools are 
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not showing significant change in suspension rates over time. Still, the random effects are significant, 
meaning there is significant variance in the model that could be accounted for by additional co-variates. 

Figure 4 shows regression lines for individual schools, colored to see the difference between PBS and 
non-PBS schools. Figure 5 shows the overall average regression lines for both categories. While these 
lines appear to have different starting points and different slopes, the differences are not large enough 
to be meaningful. 


Table 13. Elementary school model results 


Fixed Effects 

Effect 

Coefficient 

Standard Error 

T-ratio 

P-value 

Intercept 





Average Poi 

0.065 

0.030 

2.11 

0.039 

PBS Poi 

0.048 

0.036 

1.32 

0.194 

Slope 





Average Pio 

-0.003 

0.017 

-0.19 

0.85 

PBS pn 

0.036 

0.020 

1.80 

0.077 

Random Effects 

Effect 

Standard 

Deviation 

Variance 

Component 


P-value 

Intercept roi 

0.098 

0.010 

151.05 

0.000 

TIMED slope rn 

0.056 

0.003 

173.10 

0.000 

Level 1 e;t 

0.08 

0.006 
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Figure 4. Individual elementary school regression equations 

PBS schools are red where PBS=1. SUSRATE is the suspension rate for the school, that is, the number of 
suspensions divided by total enrollment for that year. TIMED denotes the year, with 0 representing 2005, 1 
representing 2006, 2 representing 2007, and 3 representing 2008. 



Figure 5. Average regression lines for PBS and non-PBS elementary schools 
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PBS schools are red where PBS=1. While these lines appear to have different starting points and different slopes, 
the differences are not large enough to be meaningful. SUSRATE is the suspension rate for the school, that is, the 
number of suspensions divided by total enrollment for that year. TIMED denotes the year, with 0 representing 
2005, 1 representing 2006, 2 representing 2007, and 3 representing 2008. 


Middle Schools 

A total of 18 middle schools fit the eligibility criteria, with 4 (22%) not implementing and 14 (78%) 
implementing PBS. The intraclass correlation coefficient, the variance in scores resulting from group 
membership, is 0.66. Again, this is a measure of how strongly suspension rates within each school are 
related to one another. The geographic characteristics of middle schools included in this analysis are 
shown in Table 14. Results for the middle school growth model are shown in Table 15. 

The average suspension rate in 2005 for non-PBS schools is significantly different from zero, yet PBS 
schools are not different from the non-PBS schools. This means that PBS and non-PBS schools do not 
have different suspension rate starting points in 2005. The fixed effect for the slope is not different from 
zero, meaning that non-PBS middle schools are not showing any significant change in suspension rates 
over these four years. The PBS contrast indicator is non-significant, meaning that the PBS schools are 
also showing no statistically significant change over time. The random effects for this model are 
significant, meaning there is more variance in the model that could be accounted for by adding 
additional co-variates. That is, other variables may explain why some schools are high and others low, or 
why some schools increase or decrease in suspensions over time. 

Figure 6 shows individual regression lines for PBS and non-PBS middle schools, with PBS schools shown 
in red. Figure 7 shows the overall regression lines for both categories. 

Table 14. Geographic characteristics of PBS and non-PBS middle schools included in analysis 



PBS 

Non-PBS 

Counties 



Kent 

5 

0 

New Castle 

7 

3 

Sussex 

2 

1 

School Districts 



Appoquinimink 

2 

0 

Caesar Rodney 

2 

0 

Capital 

2 

0 

Christina 

3 

0 

Colonial 

1 

0 

Delmar 

0 

1 

Lake Forest 

1 

0 

Red Clay 

1 

3 

Seaford 

1 

0 

Woodbridge 

1 

0 
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Table 15. Middle school model results 


Fixed Effects 

Effect 

Coefficient 

Standard Error 

T-ratio 

P-value 

Intercept 





Average Poi 

0.344 

0.148 

2.33 

0.033 

PBS poi 

0.102 

0.168 

0.611 

0.550 

Slope 





Average Pio 

0.069 

0.050 

1.38 

0.186 

PBSpn 

-0.056 

0.056 

-0.99 

0.337 

Random Effects 

Effect 

Standard 

Deviation 

Variance 

Component 


P-value 

Intercept ro. 

0.255 

0.065 

62.91 

0.000 

TIMED slope rn 

0.059 

0.003 

24.80 

0.073 

Level 1 eit 

0.178 

0.032 





PBS schools are red where PBS=1. SUSRATE is the suspension rate for the school, that is, the number of 
suspensions divided by total enrollment for that year. TIMED denotes the year, with 0 representing 2005, 1 
representing 2006, 2 representing 2007, and 3 representing 2008. 
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Figure 7. Average regression lines for PBS and non-PBS middle schools 

PBS schools are red where PBS=1. While these lines appear to have different starting points and different slopes, 
the differences are not large enough to be meaningful. 
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